[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:27 dist-test-slave-fd7l krb5kdc[28662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:27 dist-test-slave-fd7l krb5kdc[28662](info): set up 2 sockets
Aug 08 18:33:27 dist-test-slave-fd7l krb5kdc[28662](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 18:33:28.513264 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:28 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:28.644594 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42449
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:28.759699 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:28.759992 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:28.760070 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:28.764065 28676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:28.764150 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:28.764189 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:28.764246 28676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:28.764286 28676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:28.769567 28676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:28.770658 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:28.771693 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:28.780794 28676 server_base.cc:1034] running on GCE node
W20260808 18:33:28.780746 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:28.780769 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:28.780741 28685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:28.781623 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:28.782742 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:28.783911 28676 hybrid_clock.cc:648] HybridClock initialized: now 1786214008783892 us; error 102 us; skew 500 ppm
Aug 08 18:33:28 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:28.786861 28676 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:28.787746 28676 webserver.cc:460] Webserver started at http://127.27.251.254:43287/ using document root <none> and password file <none>
I20260808 18:33:28.788304 28676 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:28.788393 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:28.788671 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:28.790830 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "bffd6f8f5471472d8dfa01ee2def968f"
format_stamp: "Formatted at 2026-08-08 18:33:28 on dist-test-slave-fd7l"
server_key: "c25e32dc058e620925b9675d334c4634"
server_key_iv: "5d6eb96dbd1723bdbefb4a3d6925b170"
server_key_version: "encryptionkey@0"
I20260808 18:33:28.791399 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "bffd6f8f5471472d8dfa01ee2def968f"
format_stamp: "Formatted at 2026-08-08 18:33:28 on dist-test-slave-fd7l"
server_key: "c25e32dc058e620925b9675d334c4634"
server_key_iv: "5d6eb96dbd1723bdbefb4a3d6925b170"
server_key_version: "encryptionkey@0"
I20260808 18:33:28.795197 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:28.797801 28692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:28.799064 28676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:28.799211 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "bffd6f8f5471472d8dfa01ee2def968f"
format_stamp: "Formatted at 2026-08-08 18:33:28 on dist-test-slave-fd7l"
server_key: "c25e32dc058e620925b9675d334c4634"
server_key_iv: "5d6eb96dbd1723bdbefb4a3d6925b170"
server_key_version: "encryptionkey@0"
I20260808 18:33:28.799321 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:28.814673 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:28.817901 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:28.818118 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:28.825639 28676 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:42449
I20260808 18:33:28.825637 28746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:42449 every 8 connection(s)
I20260808 18:33:28.826669 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:28.829612 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:28.831648 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 28676
I20260808 18:33:28.831818 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:28.833513 28655 external_mini_cluster.cc:1428] Setting key e87418f62fa448230f934d7719666c1e
I20260808 18:33:28.836647 28747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: Bootstrap starting.
I20260808 18:33:28.839422 28747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:28.840361 28747 log.cc:824] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:28.842629 28747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: No bootstrap required, opened a new log
I20260808 18:33:28.847193 28747 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } }
I20260808 18:33:28.847429 28747 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:28.847520 28747 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffd6f8f5471472d8dfa01ee2def968f, State: Initialized, Role: FOLLOWER
I20260808 18:33:28.848034 28747 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } }
I20260808 18:33:28.848202 28747 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:28.848274 28747 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:28.848373 28747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:28.849474 28747 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } }
I20260808 18:33:28.849812 28747 leader_election.cc:302] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffd6f8f5471472d8dfa01ee2def968f; no voters:
I20260808 18:33:28.850134 28747 leader_election.cc:288] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:28.850266 28752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:28.850502 28752 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [term 1 LEADER]: Becoming Leader. State: Replica: bffd6f8f5471472d8dfa01ee2def968f, State: Running, Role: LEADER
I20260808 18:33:28.850837 28752 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } }
I20260808 18:33:28.851454 28747 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:28.852015 28752 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffd6f8f5471472d8dfa01ee2def968f. Latest consensus state: current_term: 1 leader_uuid: "bffd6f8f5471472d8dfa01ee2def968f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } } }
I20260808 18:33:28.852150 28752 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:28.852471 28754 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffd6f8f5471472d8dfa01ee2def968f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffd6f8f5471472d8dfa01ee2def968f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42449 } } }
I20260808 18:33:28.852609 28754 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f [sys.catalog]: This master's current role is: LEADER
Aug 08 18:33:28 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
W20260808 18:33:28.857154 28768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:28.857234 28768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:28.857352 28757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:28.858641 28757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:28.860605 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:28.837331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.27.251.254:42449)
0808 18:33:28.837813 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:28.837833 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:28.838845 (+ 1012us) server_negotiation.cc:407] Connection header received
0808 18:33:28.842954 (+ 4109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:28.842972 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:28.843265 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:28.843520 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:28.846512 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:28.847530 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:28.849283 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:28.849829 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:28.850019 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:28.855192 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:28.855198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:28.855211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:28.855237 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:28.857096 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:28.857712 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:28.857722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:28.857726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:28.857835 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:28.858175 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:28.858178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:28.858180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:28.858558 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:28.858755 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:28.859350 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 18:33:28.859615 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":138,"threads_started":1}
I20260808 18:33:28.864974 28757 catalog_manager.cc:1347] Generated new cluster ID: 4ea5aa6004a04467873ce67e221a8c59
I20260808 18:33:28.865041 28757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:28.886039 28757 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:28.887006 28757 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:28.899407 28757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffd6f8f5471472d8dfa01ee2def968f: Generated new TSK 0
I20260808 18:33:28.900218 28757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:28.947453 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:29.106329 28774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:29.106642 28774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:29.106694 28774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:29.111086 28774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:29.111146 28774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:29.111207 28774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:29.116622 28774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:29.117654 28774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:29.118660 28774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:29.128131 28783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.128127 28780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.128129 28781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:29.128479 28774 server_base.cc:1034] running on GCE node
I20260808 18:33:29.128897 28774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:29.129558 28774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:29.130741 28774 hybrid_clock.cc:648] HybridClock initialized: now 1786214009130720 us; error 31 us; skew 500 ppm
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:29.133492 28774 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:29.134330 28774 webserver.cc:460] Webserver started at http://127.27.251.193:42253/ using document root <none> and password file <none>
I20260808 18:33:29.134886 28774 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:29.134985 28774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:29.135211 28774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:29.137298 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "d802c6470d724cd4b3b54f827224f97f"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "424b52374cdae87a208a0a514ce37567"
server_key_iv: "023c248ab9e8fee37ac4ae8c7d29986b"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.137866 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "d802c6470d724cd4b3b54f827224f97f"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "424b52374cdae87a208a0a514ce37567"
server_key_iv: "023c248ab9e8fee37ac4ae8c7d29986b"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.141553 28774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:29.143906 28790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.145182 28774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:29.145310 28774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "d802c6470d724cd4b3b54f827224f97f"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "424b52374cdae87a208a0a514ce37567"
server_key_iv: "023c248ab9e8fee37ac4ae8c7d29986b"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.145418 28774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:29.167886 28774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:29.171408 28774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:29.171650 28774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:29.172324 28774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:29.173295 28774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:29.173369 28774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.173460 28774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:29.173503 28774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.183904 28774 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:36235
I20260808 18:33:29.183940 28914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:36235 every 8 connection(s)
I20260808 18:33:29.184940 28774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:29.186652 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 28774
I20260808 18:33:29.186789 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:29.187117 28655 external_mini_cluster.cc:1428] Setting key 6861781d66f0c2500aa0207b66c95f4d
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:29.200201 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.187291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:49919 (local address 127.27.251.254:42449)
0808 18:33:29.187472 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:29.187478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:29.188410 (+ 932us) server_negotiation.cc:407] Connection header received
0808 18:33:29.189806 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:29.189813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:29.189884 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:29.189988 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:29.191142 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.191696 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.192908 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.193167 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.193277 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.195814 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:29.195822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:29.195828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:29.195863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:29.197845 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.198400 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.198407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.198412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.198519 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.198878 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.198884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.198889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.199193 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:29.199316 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:29.199841 (+ 525us) server_negotiation.cc:299] Negotiation successful
0808 18:33:29.200060 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:33:29.201067 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.187745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42449 (local address 127.27.251.193:49919)
0808 18:33:29.188253 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:29.188308 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:29.189585 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:29.190124 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:29.190130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:29.190494 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:29.190990 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.191004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.191830 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.191834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.192783 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.192793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.193320 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.193324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.193498 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.194116 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:29.194133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:29.195623 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:29.197985 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.197990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.198003 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.198279 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.198627 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.198629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.198630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.198770 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.199318 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:29.199323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:29.199600 (+ 277us) client_negotiation.cc:769] Sending connection context
0808 18:33:29.199770 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 18:33:29.200110 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:29.202558 28915 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42449
I20260808 18:33:29.202889 28915 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:29.203521 28915 heartbeater.cc:511] Master 127.27.251.254:42449 requested a full tablet report, sending...
I20260808 18:33:29.205601 28709 ts_manager.cc:194] Registered new tserver with Master: d802c6470d724cd4b3b54f827224f97f (127.27.251.193:36235)
I20260808 18:33:29.206710 28709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:49919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:29.219769 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:29.336380 28921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:29.336694 28921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:29.336741 28921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:29.340718 28921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:29.340765 28921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:29.340822 28921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:29.346045 28921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:29.346987 28921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:29.347961 28921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:29.357846 28929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.357841 28926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.357839 28927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:29.357998 28921 server_base.cc:1034] running on GCE node
I20260808 18:33:29.358435 28921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:29.359033 28921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:29.360216 28921 hybrid_clock.cc:648] HybridClock initialized: now 1786214009360200 us; error 27 us; skew 500 ppm
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:29.363029 28921 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:29.363847 28921 webserver.cc:460] Webserver started at http://127.27.251.194:36785/ using document root <none> and password file <none>
I20260808 18:33:29.364396 28921 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:29.364490 28921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:29.364807 28921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:29.366773 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "1aa219713fd54f04a5fab98a4697c694"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "93f6d53f85b5efc4068e6bde88657c9c"
server_key_iv: "5e987bb333442e5ff02a01e6c193d7c4"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.367328 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "1aa219713fd54f04a5fab98a4697c694"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "93f6d53f85b5efc4068e6bde88657c9c"
server_key_iv: "5e987bb333442e5ff02a01e6c193d7c4"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.370985 28921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:33:29.373315 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.374447 28921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:29.374586 28921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "1aa219713fd54f04a5fab98a4697c694"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "93f6d53f85b5efc4068e6bde88657c9c"
server_key_iv: "5e987bb333442e5ff02a01e6c193d7c4"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.374692 28921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:29.404613 28921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:29.408023 28921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:29.408236 28921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:29.408895 28921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:29.409746 28921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:29.409814 28921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.409886 28921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:29.409953 28921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.419307 28921 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:37249
I20260808 18:33:29.419349 29059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:37249 every 8 connection(s)
I20260808 18:33:29.420207 28921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:29.426903 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 28921
I20260808 18:33:29.427018 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:29.427271 28655 external_mini_cluster.cc:1428] Setting key b9dcff15af9fc5ee2ca441f4a24f56b6
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:29.435338 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.421996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:60125 (local address 127.27.251.254:42449)
0808 18:33:29.422153 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:29.422158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:29.423025 (+ 867us) server_negotiation.cc:407] Connection header received
0808 18:33:29.424371 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:29.424375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:29.424455 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:29.424582 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:29.425775 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.426337 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.427685 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.427969 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.428079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.431163 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:29.431167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:29.431169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:29.431196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:29.432779 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.433429 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.433431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.433432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.433510 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.433987 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.433990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.433992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.434196 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:29.434276 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:29.434954 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 18:33:29.435101 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:29.436213 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.422393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42449 (local address 127.27.251.194:60125)
0808 18:33:29.422851 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:29.422915 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:29.424138 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:29.424758 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:29.424765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:29.425135 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:29.425623 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.425637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.426489 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.426494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.427552 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.427563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.428141 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.428146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.428398 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.429093 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:29.429109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:29.430922 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:29.432900 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.432909 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.432927 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.433329 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.433602 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.433608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.433613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.433873 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.434309 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:29.434319 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:29.434671 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 18:33:29.434898 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 18:33:29.435214 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260808 18:33:29.437413 29060 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42449
I20260808 18:33:29.437706 29060 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:29.438344 29060 heartbeater.cc:511] Master 127.27.251.254:42449 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:33:29.439543 28709 ts_manager.cc:194] Registered new tserver with Master: 1aa219713fd54f04a5fab98a4697c694 (127.27.251.194:37249)
I20260808 18:33:29.440099 28709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:60125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:29.459156 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:29.575953 29065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:29.576191 29065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:29.576270 29065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:29.580273 29065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:29.580324 29065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:29.580425 29065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:29.585569 29065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:29.586562 29065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:29.587561 29065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:29.597136 29072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.597167 29070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:29.597167 29074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:29.597318 29065 server_base.cc:1034] running on GCE node
I20260808 18:33:29.597877 29065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:29.598461 29065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:29.599632 29065 hybrid_clock.cc:648] HybridClock initialized: now 1786214009599617 us; error 24 us; skew 500 ppm
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:29.602502 29065 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:29.603402 29065 webserver.cc:460] Webserver started at http://127.27.251.195:45499/ using document root <none> and password file <none>
I20260808 18:33:29.603982 29065 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:29.604048 29065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:29.604221 29065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:29.606278 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "8cad1d4edd8f41fdbe85e5f4af2f432b"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "1fb0f25cd24fe9aae06eb2c624bfcbaa"
server_key_iv: "90eacf884fbf82e626d3da686dbb4319"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.606791 29065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "8cad1d4edd8f41fdbe85e5f4af2f432b"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "1fb0f25cd24fe9aae06eb2c624bfcbaa"
server_key_iv: "90eacf884fbf82e626d3da686dbb4319"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.610440 29065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:33:29.612803 29082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.614040 29065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:29.614156 29065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "8cad1d4edd8f41fdbe85e5f4af2f432b"
format_stamp: "Formatted at 2026-08-08 18:33:29 on dist-test-slave-fd7l"
server_key: "1fb0f25cd24fe9aae06eb2c624bfcbaa"
server_key_iv: "90eacf884fbf82e626d3da686dbb4319"
server_key_version: "encryptionkey@0"
I20260808 18:33:29.614255 29065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:29.641382 29065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:29.644296 29065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:29.644487 29065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:29.645125 29065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:29.646016 29065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:29.646065 29065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.646142 29065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:29.646176 29065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:29.655794 29065 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:37839
I20260808 18:33:29.655807 29204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:37839 every 8 connection(s)
I20260808 18:33:29.656739 29065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:29.666142 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29065
I20260808 18:33:29.666292 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:29.666630 28655 external_mini_cluster.cc:1428] Setting key 359ad876f865c380ca4498ec0e95e180
I20260808 18:33:29.670830 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.658542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:47587 (local address 127.27.251.254:42449)
0808 18:33:29.658687 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:29.658690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:29.659551 (+ 861us) server_negotiation.cc:407] Connection header received
0808 18:33:29.660980 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:29.660984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:29.661046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:29.661148 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:29.662332 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.662879 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.664021 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.664250 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.664357 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.667096 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:29.667099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:29.667102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:29.667126 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:29.668727 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.669300 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.669303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.669304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.669403 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.669734 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.669736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.669738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.669913 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:29.669993 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:29.670616 (+ 623us) server_negotiation.cc:299] Negotiation successful
0808 18:33:29.670714 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:33:29.671769 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.658863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42449 (local address 127.27.251.195:47587)
0808 18:33:29.659403 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:29.659445 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:29.660755 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:29.661325 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:29.661331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:29.661705 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:29.662195 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.662208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.663003 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.663007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.663895 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:29.663905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.664400 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.664404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:29.664646 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.665226 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:29.665242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:29.666918 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:29.668853 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.668859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.668874 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.669206 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.669499 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:29.669501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:29.669503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:29.669647 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:29.670003 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:29.670008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:29.670331 (+ 323us) client_negotiation.cc:769] Sending connection context
0808 18:33:29.670588 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 18:33:29.670968 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260808 18:33:29.672859 29205 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42449
I20260808 18:33:29.673107 29205 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:29.673627 29205 heartbeater.cc:511] Master 127.27.251.254:42449 requested a full tablet report, sending...
I20260808 18:33:29.674602 28709 ts_manager.cc:194] Registered new tserver with Master: 8cad1d4edd8f41fdbe85e5f4af2f432b (127.27.251.195:37839)
I20260808 18:33:29.675185 28709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:47587
I20260808 18:33:29.681088 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:29.709340 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.699437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.27.251.193:36235)
0808 18:33:29.699888 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:29.699904 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:29.699933 (+ 29us) server_negotiation.cc:407] Connection header received
0808 18:33:29.699997 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:29.700010 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:29.700212 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:29.700372 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:29.701114 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.701990 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.702921 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.703381 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.703493 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.706006 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:29.706016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:29.706034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:29.706058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:29.707503 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.707940 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.707945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.707946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.708033 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.708307 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.708311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.708312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.708617 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:29.708833 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:29.709013 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 18:33:29.709181 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
W20260808 18:33:29.710332 28820 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41820
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[28662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:29.737468 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:29.729639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.27.251.193:36235)
0808 18:33:29.729816 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:29.729820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:29.729903 (+ 83us) server_negotiation.cc:407] Connection header received
0808 18:33:29.730007 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:29.730009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:29.730056 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:29.730128 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:29.730793 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.731293 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.732085 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:29.732294 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:29.732366 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:29.734736 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:29.734739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:29.734741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:29.734764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:29.736049 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.736416 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.736419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.736420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.736495 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:29.736792 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:29.736794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:29.736795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:29.736960 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:29.737079 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:29.737257 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 18:33:29.737349 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 18:33:29.738579 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 28774
I20260808 18:33:29.747102 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 28921
I20260808 18:33:29.754391 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29065
I20260808 18:33:29.766358 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 28676
2026-08-08T18:33:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2370 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[29221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[29221](info): set up 2 sockets
Aug 08 18:33:29 dist-test-slave-fd7l krb5kdc[29221](info): commencing operation
krb5kdc: starting...
W20260808 18:33:30.866592 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:30 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:30.996085 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36283
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:31.112556 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:31.112871 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:31.112958 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:31.117193 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:31.117276 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:31.117313 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:31.117367 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:31.117405 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:31.122825 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:31.123974 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:31.125123 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:31.134181 29237 server_base.cc:1034] running on GCE node
W20260808 18:33:31.134137 29243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.134163 29244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.134141 29246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:31.135025 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:31.136216 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:31.137375 29237 hybrid_clock.cc:648] HybridClock initialized: now 1786214011137341 us; error 46 us; skew 500 ppm
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:31.140154 29237 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:31.141158 29237 webserver.cc:460] Webserver started at http://127.27.251.254:37589/ using document root <none> and password file <none>
I20260808 18:33:31.141764 29237 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:31.141827 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:31.142076 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:31.144122 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "8df68f01da90494597c62f89afa5f2ed"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "0153e060d21f230a783a518b9796e594"
server_key_iv: "84b2d6ba110206779d3417e6d4f95e46"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.144708 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "8df68f01da90494597c62f89afa5f2ed"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "0153e060d21f230a783a518b9796e594"
server_key_iv: "84b2d6ba110206779d3417e6d4f95e46"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.148355 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:31.150825 29253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.152256 29237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:31.152390 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "8df68f01da90494597c62f89afa5f2ed"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "0153e060d21f230a783a518b9796e594"
server_key_iv: "84b2d6ba110206779d3417e6d4f95e46"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.152508 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:31.166630 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:31.169584 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:31.169782 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:31.177011 29237 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:36283
I20260808 18:33:31.177037 29310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:36283 every 8 connection(s)
I20260808 18:33:31.177949 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:31.180650 29311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:31.182713 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29237
I20260808 18:33:31.182802 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:31.183068 28655 external_mini_cluster.cc:1428] Setting key 2b79ca4af835092052107ba1bdbccfbe
I20260808 18:33:31.187753 29311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed: Bootstrap starting.
I20260808 18:33:31.190474 29311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:31.191246 29311 log.cc:824] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:31.193691 29311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed: No bootstrap required, opened a new log
I20260808 18:33:31.197961 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.184461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.27.251.254:36283)
0808 18:33:31.184927 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.184949 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.184994 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:33:31.185787 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.185811 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.186232 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.186612 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:31.187252 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.188713 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.189528 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.190225 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.190404 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.193343 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:31.193347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:31.193362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:31.193388 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:31.195098 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.195578 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.195583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.195587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.195693 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.196008 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.196010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.196012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.196394 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:31.196602 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.196885 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.197210 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":107,"threads_started":1}
I20260808 18:33:31.198000 29311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } }
I20260808 18:33:31.198323 29311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:31.198398 29311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df68f01da90494597c62f89afa5f2ed, State: Initialized, Role: FOLLOWER
I20260808 18:33:31.198911 29311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } }
I20260808 18:33:31.199097 29311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:31.199172 29311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:31.199268 29311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:31.200454 29311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } }
I20260808 18:33:31.200821 29311 leader_election.cc:302] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df68f01da90494597c62f89afa5f2ed; no voters:
I20260808 18:33:31.201120 29311 leader_election.cc:288] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:31.201216 29316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:31.201450 29316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [term 1 LEADER]: Becoming Leader. State: Replica: 8df68f01da90494597c62f89afa5f2ed, State: Running, Role: LEADER
I20260808 18:33:31.201787 29316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } }
I20260808 18:33:31.202328 29311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:31.203415 29318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df68f01da90494597c62f89afa5f2ed. Latest consensus state: current_term: 1 leader_uuid: "8df68f01da90494597c62f89afa5f2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } } }
I20260808 18:33:31.203545 29318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:31.203909 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df68f01da90494597c62f89afa5f2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df68f01da90494597c62f89afa5f2ed" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36283 } } }
I20260808 18:33:31.204003 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:31.204800 29326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:31.207516 29326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:31.212776 29326 catalog_manager.cc:1347] Generated new cluster ID: 32fa0cea295f48f781e54da1ee7d0a3b
I20260808 18:33:31.212837 29326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:31.230340 29326 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:31.231272 29326 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:31.236027 29326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df68f01da90494597c62f89afa5f2ed: Generated new TSK 0
I20260808 18:33:31.236719 29326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:31.268452 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:31.384433 29338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:31.384831 29338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:31.384922 29338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:31.388968 29338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:31.389021 29338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:31.389106 29338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:31.394438 29338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:31.395463 29338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:31.396910 29338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:31.407095 29347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.407081 29344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.407096 29345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:31.407349 29338 server_base.cc:1034] running on GCE node
I20260808 18:33:31.407725 29338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:31.408289 29338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:31.409452 29338 hybrid_clock.cc:648] HybridClock initialized: now 1786214011409395 us; error 67 us; skew 500 ppm
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:31.412334 29338 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:31.413257 29338 webserver.cc:460] Webserver started at http://127.27.251.193:35869/ using document root <none> and password file <none>
I20260808 18:33:31.413784 29338 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:31.413843 29338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:31.414011 29338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:31.416186 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "8a46872cf43e48a49ff3a653c8abb9e3"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "94e429396a9f786849870877d23270f9"
server_key_iv: "ae6b86b0b645a7627868c51150728005"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.416782 29338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "8a46872cf43e48a49ff3a653c8abb9e3"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "94e429396a9f786849870877d23270f9"
server_key_iv: "ae6b86b0b645a7627868c51150728005"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.420280 29338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:33:31.422794 29355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.424005 29338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:31.424139 29338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "8a46872cf43e48a49ff3a653c8abb9e3"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "94e429396a9f786849870877d23270f9"
server_key_iv: "ae6b86b0b645a7627868c51150728005"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.424248 29338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:31.444810 29338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:31.448047 29338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:31.448246 29338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:31.448900 29338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:31.449741 29338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:31.449808 29338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.449875 29338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:31.449921 29338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.459486 29338 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:33125
I20260808 18:33:31.459506 29477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:33125 every 8 connection(s)
I20260808 18:33:31.460389 29338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:31.466162 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29338
I20260808 18:33:31.466274 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:31.466589 28655 external_mini_cluster.cc:1428] Setting key bece031340b5524263ad225df8185ad3
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:31.476771 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.462301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:44923 (local address 127.27.251.254:36283)
0808 18:33:31.462478 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.462484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.463391 (+ 907us) server_negotiation.cc:407] Connection header received
0808 18:33:31.465395 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.465399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.465481 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.465590 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:31.467359 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.468188 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.469938 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.470230 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.470344 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.473124 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:31.473128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:31.473131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:31.473192 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:31.474621 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.475137 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.475141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.475143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.475228 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.475614 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.475618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.475619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.475792 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:31.475898 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.476424 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.476592 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:33:31.477587 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.462699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36283 (local address 127.27.251.193:44923)
0808 18:33:31.463222 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.463286 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.465124 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.465804 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.465815 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.466417 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:31.467185 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.467207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.468321 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.468325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.469810 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.469820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.470391 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.470395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.470578 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.471189 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:31.471206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:31.472950 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:31.474736 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.474742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.474758 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.475022 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.475317 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.475321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.475323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.475464 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.475898 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:31.475917 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:31.476182 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.476367 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.476733 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:31.479019 29479 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36283
I20260808 18:33:31.479264 29479 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:31.479792 29479 heartbeater.cc:511] Master 127.27.251.254:36283 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:31.481410 29271 ts_manager.cc:194] Registered new tserver with Master: 8a46872cf43e48a49ff3a653c8abb9e3 (127.27.251.193:33125)
I20260808 18:33:31.482543 29271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:44923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:31.497337 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:31.616070 29484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:31.616328 29484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:31.616403 29484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:31.620492 29484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:31.620617 29484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:31.620716 29484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:31.626116 29484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:31.627192 29484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:31.628300 29484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:31.638206 29489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.638249 29490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.638253 29492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:31.638595 29484 server_base.cc:1034] running on GCE node
I20260808 18:33:31.639003 29484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:31.639663 29484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:31.640839 29484 hybrid_clock.cc:648] HybridClock initialized: now 1786214011640804 us; error 46 us; skew 500 ppm
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:31.643608 29484 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:31.644454 29484 webserver.cc:460] Webserver started at http://127.27.251.194:35743/ using document root <none> and password file <none>
I20260808 18:33:31.645049 29484 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:31.645152 29484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:31.645385 29484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:31.647418 29484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "74eed3a188fc4dee93e07e4f1cd96aec"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "528adfcd2bf40a749a2f57883feaf7db"
server_key_iv: "b07b87b0e2ce5f691855299ad0e50691"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.647980 29484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "74eed3a188fc4dee93e07e4f1cd96aec"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "528adfcd2bf40a749a2f57883feaf7db"
server_key_iv: "b07b87b0e2ce5f691855299ad0e50691"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.651752 29484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:33:31.654382 29500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.655599 29484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:31.655750 29484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "74eed3a188fc4dee93e07e4f1cd96aec"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "528adfcd2bf40a749a2f57883feaf7db"
server_key_iv: "b07b87b0e2ce5f691855299ad0e50691"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.655894 29484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:31.669368 29484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:31.672870 29484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:31.673086 29484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:31.673754 29484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:31.674611 29484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:31.674676 29484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.674741 29484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:31.674780 29484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.683434 29484 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:33281
I20260808 18:33:31.683459 29629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:33281 every 8 connection(s)
I20260808 18:33:31.684336 29484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:31.693784 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29484
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:31.693882 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:31.694154 28655 external_mini_cluster.cc:1428] Setting key 78a0f5e701de205eb0057da215c0ddf1
I20260808 18:33:31.698465 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.686141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:39985 (local address 127.27.251.254:36283)
0808 18:33:31.686316 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.686321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.687153 (+ 832us) server_negotiation.cc:407] Connection header received
0808 18:33:31.688568 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.688574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.688644 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.688725 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:31.689914 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.690469 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.691690 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.691903 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.691988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.694517 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:31.694520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:31.694522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:31.694545 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:31.695960 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.696498 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.696542 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.696545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.696636 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.696975 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.696980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.696982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.697246 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:31.697354 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.698106 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.698248 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 18:33:31.699258 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.686545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36283 (local address 127.27.251.194:39985)
0808 18:33:31.686984 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.687039 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.688282 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.688890 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.688898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.689272 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:31.689755 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.689769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.690593 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.690595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.691579 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.691589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.692061 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.692064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.692228 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.692830 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:31.692846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:31.694347 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:31.696096 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.696102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.696116 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.696384 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.696727 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.696730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.696732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.696879 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.697417 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:31.697425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:31.697784 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.698029 (+ 245us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.698350 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260808 18:33:31.700466 29630 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36283
I20260808 18:33:31.700757 29630 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:31.701256 29630 heartbeater.cc:511] Master 127.27.251.254:36283 requested a full tablet report, sending...
I20260808 18:33:31.702418 29271 ts_manager.cc:194] Registered new tserver with Master: 74eed3a188fc4dee93e07e4f1cd96aec (127.27.251.194:33281)
I20260808 18:33:31.702992 29271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:39985
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:31.723621 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:31.842319 29636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:31.842610 29636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:31.842693 29636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:31.847289 29636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:31.847357 29636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:31.847476 29636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:31.853571 29636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:31.855050 29636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:31.856272 29636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:31.867349 29644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:31.867328 29641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:31.867523 29636 server_base.cc:1034] running on GCE node
W20260808 18:33:31.867671 29642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:31.868111 29636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:31.868785 29636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:31.869946 29636 hybrid_clock.cc:648] HybridClock initialized: now 1786214011869913 us; error 43 us; skew 500 ppm
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:31.872792 29636 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:31.873678 29636 webserver.cc:460] Webserver started at http://127.27.251.195:42347/ using document root <none> and password file <none>
I20260808 18:33:31.874250 29636 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:31.874339 29636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:31.874577 29636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:31.876688 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "09aa82e0f15843628be08f3ea4bddbe9"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "6dddb66c79b78a6b88ef8fedf379603e"
server_key_iv: "cdec236c2159302ecbb35b1e6e6e729e"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.877274 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "09aa82e0f15843628be08f3ea4bddbe9"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "6dddb66c79b78a6b88ef8fedf379603e"
server_key_iv: "cdec236c2159302ecbb35b1e6e6e729e"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.880961 29636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 18:33:31.883392 29651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.884622 29636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:31.884747 29636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "09aa82e0f15843628be08f3ea4bddbe9"
format_stamp: "Formatted at 2026-08-08 18:33:31 on dist-test-slave-fd7l"
server_key: "6dddb66c79b78a6b88ef8fedf379603e"
server_key_iv: "cdec236c2159302ecbb35b1e6e6e729e"
server_key_version: "encryptionkey@0"
I20260808 18:33:31.884851 29636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:31.896585 29636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:31.899801 29636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:31.900002 29636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:31.900686 29636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:31.901528 29636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:31.901605 29636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.901690 29636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:31.901737 29636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:31.911666 29636 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:41813
I20260808 18:33:31.911708 29775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:41813 every 8 connection(s)
I20260808 18:33:31.912729 29636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:31.920274 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29636
I20260808 18:33:31.920411 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:31.920761 28655 external_mini_cluster.cc:1428] Setting key 47f79c46539da041a2c5a5c7d9534a14
Aug 08 18:33:31 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:31.927261 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.914651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:48647 (local address 127.27.251.254:36283)
0808 18:33:31.914788 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.914792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.915843 (+ 1051us) server_negotiation.cc:407] Connection header received
0808 18:33:31.917282 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.917286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.917356 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.917444 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:31.918631 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.919171 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.920494 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.920860 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.920945 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.923891 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:31.923899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:31.923901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:31.923925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:31.925228 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.925742 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.925746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.925747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.925820 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.926155 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.926159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.926160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.926346 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:31.926457 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.926955 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.927091 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:33:31.928151 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.915109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36283 (local address 127.27.251.195:48647)
0808 18:33:31.915683 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.915728 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.916994 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.917637 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.917643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.918003 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:31.918479 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.918492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.919303 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.919306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.920332 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.920347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.921162 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.921165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.921355 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.922057 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:31.922111 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:31.923735 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:31.925343 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.925349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.925364 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.925629 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.925911 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:31.925914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:31.925916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:31.926059 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:31.926456 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:31.926463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:31.926724 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.926913 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.927237 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":158,"threads_started":1}
I20260808 18:33:31.929423 29776 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36283
I20260808 18:33:31.929646 29776 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:31.930085 29776 heartbeater.cc:511] Master 127.27.251.254:36283 requested a full tablet report, sending...
I20260808 18:33:31.931035 29271 ts_manager.cc:194] Registered new tserver with Master: 09aa82e0f15843628be08f3ea4bddbe9 (127.27.251.195:41813)
I20260808 18:33:31.931610 29271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:48647
I20260808 18:33:31.934640 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:33:31.945708 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.938471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.27.251.254:36283)
0808 18:33:31.938627 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.938632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.938846 (+ 214us) server_negotiation.cc:407] Connection header received
0808 18:33:31.939021 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.939024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.939070 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.939145 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:31.939831 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.940332 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.941170 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.941380 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.941461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.942822 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:31.942825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:31.942827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:31.942848 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:31.944087 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.944452 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.944454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.944455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.944608 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:31.944854 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:31.944856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:31.944857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:31.945077 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:31.945181 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.945377 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.945478 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:33:31.950771 29271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:31.954057 29271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:31.970062 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.963500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:33281 (local address 127.0.0.1:41834)
0808 18:33:31.964335 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.964387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.964560 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.965883 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.965889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.965912 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:31.966118 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.966126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967174 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.967180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.968451 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.968460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.969438 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.969442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.969578 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.969625 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.969742 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.969882 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":190,"thread_start_us":107,"threads_started":1}
I20260808 18:33:31.970089 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.963618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.27.251.194:33281)
0808 18:33:31.965146 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.965167 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.965195 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:33:31.965286 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.965292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.965532 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.965693 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:31.966253 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967033 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.968624 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.969304 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.969435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.969582 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.969809 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.969914 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":121,"threads_started":1}
I20260808 18:33:31.970685 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.963615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:41813 (local address 127.0.0.1:34060)
0808 18:33:31.965005 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.965022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.965138 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.965596 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.965606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.965638 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:31.965803 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.965811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967514 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.967518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.968947 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.968959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.970299 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.970303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.970397 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.970426 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.970494 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.970562 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":105,"threads_started":1}
I20260808 18:33:31.971268 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.964097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:33125 (local address 127.0.0.1:55752)
0808 18:33:31.965338 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:31.965352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:31.965446 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:31.965923 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:31.965926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:31.965947 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:31.966139 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.966145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967722 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.967725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.969052 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:31.969064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.970786 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.970791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:31.970912 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.970944 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 18:33:31.971054 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 18:33:31.971133 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":31,"thread_start_us":99,"threads_started":1}
I20260808 18:33:31.971282 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.964083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.27.251.195:41813)
0808 18:33:31.964642 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.964657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.965145 (+ 488us) server_negotiation.cc:407] Connection header received
0808 18:33:31.965242 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.965246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.965435 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.965594 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:31.965960 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967341 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.969114 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.969954 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.970086 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.970972 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.971026 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.971139 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":183,"threads_started":1}
I20260808 18:33:31.971655 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:31.964377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.27.251.193:33125)
0808 18:33:31.964813 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:31.964830 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:31.965451 (+ 621us) server_negotiation.cc:407] Connection header received
0808 18:33:31.965594 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:31.965599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:31.965771 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:31.965921 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:31.966323 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.967567 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.969187 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:31.970054 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:31.970591 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:31.970697 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:31.971094 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 18:33:31.971197 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":136,"threads_started":1}
I20260808 18:33:31.973754 29555 tablet_service.cc:1467] Processing CreateTablet for tablet 536b0a09b6164f4dbd070ebc82103a3d (DEFAULT_TABLE table=test-table [id=dc5c9641c4484e87853a12db90b43ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:31.974129 29704 tablet_service.cc:1467] Processing CreateTablet for tablet 536b0a09b6164f4dbd070ebc82103a3d (DEFAULT_TABLE table=test-table [id=dc5c9641c4484e87853a12db90b43ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:31.974249 29408 tablet_service.cc:1467] Processing CreateTablet for tablet 536b0a09b6164f4dbd070ebc82103a3d (DEFAULT_TABLE table=test-table [id=dc5c9641c4484e87853a12db90b43ac6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:31.974995 29555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536b0a09b6164f4dbd070ebc82103a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:31.975131 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536b0a09b6164f4dbd070ebc82103a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:31.975435 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536b0a09b6164f4dbd070ebc82103a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:31.982133 29795 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: Bootstrap starting.
I20260808 18:33:31.982900 29799 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Bootstrap starting.
I20260808 18:33:31.983425 29796 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: Bootstrap starting.
I20260808 18:33:31.985278 29799 tablet_bootstrap.cc:654] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:31.985278 29795 tablet_bootstrap.cc:654] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:31.985621 29796 tablet_bootstrap.cc:654] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:31.986510 29796 log.cc:824] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:31.986557 29799 log.cc:824] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:31.986601 29795 log.cc:824] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:31.988924 29795 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: No bootstrap required, opened a new log
I20260808 18:33:31.988924 29796 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: No bootstrap required, opened a new log
I20260808 18:33:31.989151 29796 ts_tablet_manager.cc:1397] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:31.989151 29795 ts_tablet_manager.cc:1397] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:33:31.993968 29799 tablet_bootstrap.cc:492] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: No bootstrap required, opened a new log
I20260808 18:33:31.993754 29796 raft_consensus.cc:348] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.994077 29796 raft_consensus.cc:374] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:31.994149 29796 raft_consensus.cc:729] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74eed3a188fc4dee93e07e4f1cd96aec, State: Initialized, Role: FOLLOWER
I20260808 18:33:31.994207 29799 ts_tablet_manager.cc:1397] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260808 18:33:31.994113 29795 raft_consensus.cc:348] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.994382 29795 raft_consensus.cc:374] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:31.994445 29795 raft_consensus.cc:729] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a46872cf43e48a49ff3a653c8abb9e3, State: Initialized, Role: FOLLOWER
I20260808 18:33:31.994683 29796 consensus_queue.cc:260] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.994874 29795 consensus_queue.cc:260] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.995472 29630 heartbeater.cc:503] Master 127.27.251.254:36283 was elected leader, sending a full tablet report...
I20260808 18:33:31.995850 29795 ts_tablet_manager.cc:1428] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 18:33:31.995850 29796 ts_tablet_manager.cc:1428] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:33:31.996210 29479 heartbeater.cc:503] Master 127.27.251.254:36283 was elected leader, sending a full tablet report...
I20260808 18:33:31.998405 29799 raft_consensus.cc:348] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.998713 29799 raft_consensus.cc:374] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:31.998772 29799 raft_consensus.cc:729] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09aa82e0f15843628be08f3ea4bddbe9, State: Initialized, Role: FOLLOWER
I20260808 18:33:31.999224 29799 consensus_queue.cc:260] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:31.999872 29776 heartbeater.cc:503] Master 127.27.251.254:36283 was elected leader, sending a full tablet report...
I20260808 18:33:32.000281 29799 ts_tablet_manager.cc:1428] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:33:32.055663 29805 raft_consensus.cc:482] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:32.055917 29805 raft_consensus.cc:504] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:32.057200 29805 leader_election.cc:288] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74eed3a188fc4dee93e07e4f1cd96aec (127.27.251.194:33281), 8a46872cf43e48a49ff3a653c8abb9e3 (127.27.251.193:33125)
I20260808 18:33:32.061600 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.057641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:59549 (local address 127.27.251.194:33281)
0808 18:33:32.057777 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:32.057781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:32.057799 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:33:32.057873 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:32.057876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:32.057924 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:32.058015 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:32.058351 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.058964 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.060614 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.061018 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.061153 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.061211 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:32.061390 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 18:33:32.061459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 18:33:32.061602 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.057440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:33281 (local address 127.27.251.195:59549)
0808 18:33:32.057655 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:32.057671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:32.057771 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:32.058024 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:32.058047 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:32.058098 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:32.058228 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:32.058234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.059080 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.059084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:32.060460 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:32.060471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.061141 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.061146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:32.061257 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.061280 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:33:32.061372 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:33:32.061450 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":4,"spinlock_wait_cycles":17408}
I20260808 18:33:32.061847 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.057610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:53655 (local address 127.27.251.193:33125)
0808 18:33:32.057745 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:32.057749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:32.058040 (+ 291us) server_negotiation.cc:407] Connection header received
0808 18:33:32.058217 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:32.058224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:32.058271 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:32.058349 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:32.058628 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.059359 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.060830 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.061251 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.061381 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.061438 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:32.061659 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 18:33:32.061739 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:33:32.061864 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.057489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:33125 (local address 127.27.251.195:53655)
0808 18:33:32.057936 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:32.057947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:32.058107 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:32.058370 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:32.058373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:32.058390 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:32.058513 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:32.058519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.059504 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.059507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:32.060711 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:32.060719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.061373 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.061377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:32.061484 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.061546 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 18:33:32.061638 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:33:32.061706 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":51,"spinlock_wait_cycles":16384,"thread_start_us":156,"threads_started":1}
I20260808 18:33:32.062466 29429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536b0a09b6164f4dbd070ebc82103a3d" candidate_uuid: "09aa82e0f15843628be08f3ea4bddbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" is_pre_election: true
I20260808 18:33:32.062459 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536b0a09b6164f4dbd070ebc82103a3d" candidate_uuid: "09aa82e0f15843628be08f3ea4bddbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" is_pre_election: true
I20260808 18:33:32.062774 29429 raft_consensus.cc:2454] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09aa82e0f15843628be08f3ea4bddbe9 in term 0.
I20260808 18:33:32.062779 29577 raft_consensus.cc:2454] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09aa82e0f15843628be08f3ea4bddbe9 in term 0.
I20260808 18:33:32.063304 29655 leader_election.cc:302] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09aa82e0f15843628be08f3ea4bddbe9, 74eed3a188fc4dee93e07e4f1cd96aec, 8a46872cf43e48a49ff3a653c8abb9e3; no voters:
I20260808 18:33:32.063611 29805 raft_consensus.cc:2781] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:32.063697 29805 raft_consensus.cc:482] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:32.063740 29805 raft_consensus.cc:3037] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:32.064829 29805 raft_consensus.cc:504] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:32.065167 29805 leader_election.cc:288] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [CANDIDATE]: Term 1 election: Requested vote from peers 74eed3a188fc4dee93e07e4f1cd96aec (127.27.251.194:33281), 8a46872cf43e48a49ff3a653c8abb9e3 (127.27.251.193:33125)
I20260808 18:33:32.065511 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536b0a09b6164f4dbd070ebc82103a3d" candidate_uuid: "09aa82e0f15843628be08f3ea4bddbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74eed3a188fc4dee93e07e4f1cd96aec"
I20260808 18:33:32.065511 29429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536b0a09b6164f4dbd070ebc82103a3d" candidate_uuid: "09aa82e0f15843628be08f3ea4bddbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a46872cf43e48a49ff3a653c8abb9e3"
I20260808 18:33:32.065642 29577 raft_consensus.cc:3037] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:32.065644 29429 raft_consensus.cc:3037] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:32.067001 29577 raft_consensus.cc:2454] T 536b0a09b6164f4dbd070ebc82103a3d P 74eed3a188fc4dee93e07e4f1cd96aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09aa82e0f15843628be08f3ea4bddbe9 in term 1.
I20260808 18:33:32.067001 29429 raft_consensus.cc:2454] T 536b0a09b6164f4dbd070ebc82103a3d P 8a46872cf43e48a49ff3a653c8abb9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09aa82e0f15843628be08f3ea4bddbe9 in term 1.
I20260808 18:33:32.067312 29655 leader_election.cc:302] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09aa82e0f15843628be08f3ea4bddbe9, 8a46872cf43e48a49ff3a653c8abb9e3; no voters:
I20260808 18:33:32.067474 29805 raft_consensus.cc:2781] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:32.067706 29805 raft_consensus.cc:686] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [term 1 LEADER]: Becoming Leader. State: Replica: 09aa82e0f15843628be08f3ea4bddbe9, State: Running, Role: LEADER
I20260808 18:33:32.068068 29805 consensus_queue.cc:237] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } }
I20260808 18:33:32.071182 29271 catalog_manager.cc:5495] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09aa82e0f15843628be08f3ea4bddbe9 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "09aa82e0f15843628be08f3ea4bddbe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74eed3a188fc4dee93e07e4f1cd96aec" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09aa82e0f15843628be08f3ea4bddbe9" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 41813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a46872cf43e48a49ff3a653c8abb9e3" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 33125 } health_report { overall_health: UNKNOWN } } }
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:32.094058 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.085138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.27.251.193:33125)
0808 18:33:32.085289 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:32.085293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:32.085470 (+ 177us) server_negotiation.cc:407] Connection header received
0808 18:33:32.085598 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:32.085601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:32.085648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:32.085711 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:32.086383 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.086879 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.087681 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.087967 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.088039 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.090528 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:32.090540 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:32.090555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:32.090581 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:32.092372 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.092808 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.092814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.092815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.092898 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.093141 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.093144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.093145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.093389 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:32.093562 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:32.093697 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 18:33:32.093803 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:32.123274 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.114574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.27.251.193:33125)
0808 18:33:32.114752 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:32.114758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:32.114892 (+ 134us) server_negotiation.cc:407] Connection header received
0808 18:33:32.115058 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:32.115062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:32.115122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:32.115209 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:32.115967 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.116605 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.117453 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.117715 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.117824 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.120163 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:32.120172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:32.120174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:32.120198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:32.121491 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.121944 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.121949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.121951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.122034 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.122305 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.122309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.122311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.122547 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:32.122684 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:32.122971 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 18:33:32.123105 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:32.150542 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:32.142044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.27.251.193:33125)
0808 18:33:32.142240 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:32.142245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:32.142402 (+ 157us) server_negotiation.cc:407] Connection header received
0808 18:33:32.142568 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:32.142572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:32.142624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:32.142716 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:32.143425 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.143954 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.144890 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:32.145112 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:32.145233 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:32.147532 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:32.147536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:32.147538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:32.147562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:32.148898 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.149323 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.149327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.149328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.149413 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:32.149687 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:32.149690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:32.149692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:32.149870 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:32.150008 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:32.150240 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 18:33:32.150368 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 18:33:32.152112 29386 tablet_service.cc:2908] Rejecting scan request for tablet 536b0a09b6164f4dbd070ebc82103a3d: Uninitialized: clean time has not yet been initialized
I20260808 18:33:32.154495 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29338
W20260808 18:33:32.163837 29777 tablet.cc:2369] T 536b0a09b6164f4dbd070ebc82103a3d P 09aa82e0f15843628be08f3ea4bddbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:32.164208 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29484
I20260808 18:33:32.172642 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29636
I20260808 18:33:32.179875 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29237
2026-08-08T18:33:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2414 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29823](info): set up 2 sockets
Aug 08 18:33:32 dist-test-slave-fd7l krb5kdc[29823](info): commencing operation
krb5kdc: starting...
W20260808 18:33:33.298128 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:33 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:33.427841 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43293
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:43293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:33.543182 29841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:33.543452 29841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:33.543502 29841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:33.547492 29841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:33.547554 29841 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:33:33.547571 29841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:33.547590 29841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:33.547609 29841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:33.547626 29841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:33.553108 29841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:43293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:33.554231 29841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:33.555258 29841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:33.563910 29847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:33.563928 29849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:33.564008 29846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:33.564145 29841 server_base.cc:1034] running on GCE node
I20260808 18:33:33.564903 29841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:33.565979 29841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:33.567134 29841 hybrid_clock.cc:648] HybridClock initialized: now 1786214013567107 us; error 39 us; skew 500 ppm
Aug 08 18:33:33 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:33.573720 29841 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:33.574645 29841 webserver.cc:460] Webserver started at http://127.27.251.254:33067/ using document root <none> and password file <none>
I20260808 18:33:33.575212 29841 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:33.575304 29841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:33.575532 29841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:33.577643 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "9eb5237ee4274cd7b094ce0b5b8f3477"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "fe5db496073bb328cea84aff715a5dcf"
server_key_iv: "4797ca2b620a9ff2a2d2fc3c2d5aa469"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.578218 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "9eb5237ee4274cd7b094ce0b5b8f3477"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "fe5db496073bb328cea84aff715a5dcf"
server_key_iv: "4797ca2b620a9ff2a2d2fc3c2d5aa469"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.581898 29841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:33.584295 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:33.585604 29841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:33.585741 29841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "9eb5237ee4274cd7b094ce0b5b8f3477"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "fe5db496073bb328cea84aff715a5dcf"
server_key_iv: "4797ca2b620a9ff2a2d2fc3c2d5aa469"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.585847 29841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:33.595788 29841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:33.599007 29841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:33.599210 29841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:33.607679 29841 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:43293
I20260808 18:33:33.607734 29924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:43293 every 8 connection(s)
I20260808 18:33:33.608700 29841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:33.612082 29925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:33.614027 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29841
I20260808 18:33:33.614161 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:33.614435 28655 external_mini_cluster.cc:1428] Setting key d4779ebc2d119902e48260d55b7077e5
I20260808 18:33:33.619017 29925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Bootstrap starting.
I20260808 18:33:33.621764 29925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:33.622746 29925 log.cc:824] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:33 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:33.624827 29925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: No bootstrap required, opened a new log
I20260808 18:33:33.628232 29925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } }
I20260808 18:33:33.628448 29925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:33.628506 29925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb5237ee4274cd7b094ce0b5b8f3477, State: Initialized, Role: FOLLOWER
I20260808 18:33:33.629045 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:33.615891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.27.251.254:43293)
0808 18:33:33.616287 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:33.616311 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:33.616347 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:33:33.617034 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:33.617052 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:33.617386 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:33.617682 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:33.618620 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.619682 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.620917 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.621391 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.621512 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:33.624362 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:33.624367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:33.624381 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:33.624408 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:33.626196 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.626653 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.626657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.626662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.626769 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.627056 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.627060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.627062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.627437 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:33.627676 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:33.627909 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 18:33:33.628187 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
I20260808 18:33:33.628989 29925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } }
I20260808 18:33:33.629201 29925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:33.629299 29925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:33.629428 29925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:33.630412 29925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } }
I20260808 18:33:33.630780 29925 leader_election.cc:302] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eb5237ee4274cd7b094ce0b5b8f3477; no voters:
I20260808 18:33:33.631110 29925 leader_election.cc:288] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:33.631212 29930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:33.631517 29930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [term 1 LEADER]: Becoming Leader. State: Replica: 9eb5237ee4274cd7b094ce0b5b8f3477, State: Running, Role: LEADER
I20260808 18:33:33.631816 29930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } }
I20260808 18:33:33.632404 29925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:33.633575 29932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } } }
I20260808 18:33:33.633575 29931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eb5237ee4274cd7b094ce0b5b8f3477. Latest consensus state: current_term: 1 leader_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb5237ee4274cd7b094ce0b5b8f3477" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43293 } } }
I20260808 18:33:33.633766 29931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:33.634069 29932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:33.634153 29940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:33.637413 29940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:33.642282 29940 catalog_manager.cc:1347] Generated new cluster ID: 8b15949c732642e1be18691429a01a3f
I20260808 18:33:33.642347 29940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:33.653285 29940 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:33.654477 29940 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:33.660168 29940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Generated new TSK 0
I20260808 18:33:33.661000 29940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:33.700793 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:43293
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:33.817886 29954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:33.818163 29954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:33.818236 29954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:33.822659 29954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:33.822757 29954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:33.822834 29954 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:33.822912 29954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:33.828341 29954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:33.829424 29954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:33.830449 29954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:33.840806 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:33.840837 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:33.840947 29954 server_base.cc:1034] running on GCE node
W20260808 18:33:33.840822 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:33.841437 29954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:33.842231 29954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:33.843532 29954 hybrid_clock.cc:648] HybridClock initialized: now 1786214013843511 us; error 30 us; skew 500 ppm
Aug 08 18:33:33 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:33.846597 29954 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:33.847520 29954 webserver.cc:460] Webserver started at http://127.27.251.193:42777/ using document root <none> and password file <none>
I20260808 18:33:33.848102 29954 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:33.848197 29954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:33.848428 29954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:33.850837 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "c6217e607afc41feaa98019c4bafcf80"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "22c68bddcbee396d1f8ab05749ac0d65"
server_key_iv: "191f72c5e58eb7114783bddebd39f0ce"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.851492 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "c6217e607afc41feaa98019c4bafcf80"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "22c68bddcbee396d1f8ab05749ac0d65"
server_key_iv: "191f72c5e58eb7114783bddebd39f0ce"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.855406 29954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:33:33.857903 29970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:33.859213 29954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:33.859356 29954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c6217e607afc41feaa98019c4bafcf80"
format_stamp: "Formatted at 2026-08-08 18:33:33 on dist-test-slave-fd7l"
server_key: "22c68bddcbee396d1f8ab05749ac0d65"
server_key_iv: "191f72c5e58eb7114783bddebd39f0ce"
server_key_version: "encryptionkey@0"
I20260808 18:33:33.859467 29954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:33.878688 29954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:33.881644 29954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:33.881955 29954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:33.883809 29954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:33.883880 29954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:33.883947 29954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:33.884006 29954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:33 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:33.895663 29954 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:42623
I20260808 18:33:33.895725 30101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:42623 every 8 connection(s)
I20260808 18:33:33.896785 29954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:33.898265 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 29954
I20260808 18:33:33.898360 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:33.898593 28655 external_mini_cluster.cc:1428] Setting key 08eca1f7e1c4134735a09a7d6386274f
I20260808 18:33:33.899137 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:33.884280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:36627 (local address 127.27.251.254:43293)
0808 18:33:33.884408 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:33.884412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:33.885587 (+ 1175us) server_negotiation.cc:407] Connection header received
0808 18:33:33.886561 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:33.886573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:33.886634 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:33.886739 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:33.888335 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.889233 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.890816 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.891221 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.891346 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:33.894916 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:33.894922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:33.894924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:33.894954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:33.896658 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.897230 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.897233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.897235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.897327 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.897894 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.897899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.897900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.898131 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:33.898248 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:33.898813 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 18:33:33.898950 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:33:33.899972 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:33.884896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.193:36627)
0808 18:33:33.885409 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:33.885452 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:33.886320 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:33.886930 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:33.886950 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:33.887467 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:33.888172 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:33.888192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.889368 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.889372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:33.890691 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:33.890702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.891408 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.891413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:33.891618 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:33.892255 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:33.892282 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:33.894746 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:33.896781 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:33.896787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:33.896801 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:33.897118 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:33.897435 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:33.897438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:33.897561 (+ 123us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:33.897779 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:33.898249 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:33.898256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:33.898561 (+ 305us) client_negotiation.cc:769] Sending connection context
0808 18:33:33.898766 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 18:33:33.899083 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260808 18:33:33.901450 30102 heartbeater.cc:347] Connected to a master server at 127.27.251.254:43293
I20260808 18:33:33.901611 30102 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:33.902222 30102 heartbeater.cc:511] Master 127.27.251.254:43293 requested a full tablet report, sending...
I20260808 18:33:33.904073 29874 ts_manager.cc:194] Registered new tserver with Master: c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623)
I20260808 18:33:33.905472 29874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:36627
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:33.911950 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:33.903988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:58769 (local address 127.27.251.254:43293)
0808 18:33:33.904099 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:33.904103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:33.904300 (+ 197us) server_negotiation.cc:407] Connection header received
0808 18:33:33.904396 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:33.904399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:33.904449 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:33.904554 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:33.905207 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.905925 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.907303 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.907531 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.907642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:33.908842 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:33.908846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:33.908849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:33.908873 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:33.910294 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.910728 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.910731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.910733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.910841 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:33.911121 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:33.911124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:33.911126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:33.911300 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:33.911409 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:33.911678 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 18:33:33.911787 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260808 18:33:33.911957 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:33.903906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.193:58769)
0808 18:33:33.904210 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:33.904227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:33.904322 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:33.904634 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:33.904640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:33.904869 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:33.905031 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:33.905042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.906079 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.906085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:33.907150 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:33.907184 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:33.907647 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:33.907651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:33.907741 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:33.908271 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:33.908288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:33.908706 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:33.910410 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:33.910413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:33.910415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:33.910622 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:33.910923 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:33.910927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:33.910929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:33.911018 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:33.911401 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:33.911404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:33.911504 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 18:33:33.911621 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:33:33.911762 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:33.932149 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:43293
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:34.050669 30113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:34.050915 30113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:34.050959 30113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:34.055263 30113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:34.055327 30113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:34.055368 30113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:34.055399 30113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:34.060652 30113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:34.061666 30113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:34.062716 30113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:34.072655 30118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:34.072638 30121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:34.072749 30113 server_base.cc:1034] running on GCE node
W20260808 18:33:34.072638 30119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:34.073320 30113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:34.074008 30113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:34.075197 30113 hybrid_clock.cc:648] HybridClock initialized: now 1786214014075178 us; error 31 us; skew 500 ppm
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:34.078104 30113 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:34.079047 30113 webserver.cc:460] Webserver started at http://127.27.251.194:41639/ using document root <none> and password file <none>
I20260808 18:33:34.079607 30113 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:34.079701 30113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:34.079939 30113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:34.082129 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "bf5258f7bed54566b5d4ff63102f32a1"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "a16b38e720612bd9fb8121329680ffb7"
server_key_iv: "61630178ae3e095c16dd4b117cea8544"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.082733 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "bf5258f7bed54566b5d4ff63102f32a1"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "a16b38e720612bd9fb8121329680ffb7"
server_key_iv: "61630178ae3e095c16dd4b117cea8544"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.086696 30113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:34.089308 30129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:34.090608 30113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:34.090737 30113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "bf5258f7bed54566b5d4ff63102f32a1"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "a16b38e720612bd9fb8121329680ffb7"
server_key_iv: "61630178ae3e095c16dd4b117cea8544"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.090847 30113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:34.131498 30113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:34.135418 30113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:34.135630 30113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:34.137221 30113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:34.137320 30113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:34.137408 30113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:34.137465 30113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:34.148623 30113 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:38249
I20260808 18:33:34.148685 30257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:38249 every 8 connection(s)
I20260808 18:33:34.149888 30113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:34.151959 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.137956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36269 (local address 127.27.251.254:43293)
0808 18:33:34.138132 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.138138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.139606 (+ 1468us) server_negotiation.cc:407] Connection header received
0808 18:33:34.141096 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.141100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.141163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.141260 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.142681 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.143229 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.144642 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.144872 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.144993 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.147805 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.147813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.147817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.147852 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.149764 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.150451 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.150454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.150456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.150548 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.150953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.150956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.150958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.151172 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.151256 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.151715 (+ 459us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.151836 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:34.152864 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.138891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.194:36269)
0808 18:33:34.139441 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.139498 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.140847 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.141644 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.141656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.142009 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:34.142511 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.142525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.143364 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.143367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.144458 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.144468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.145038 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.145042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.145231 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.145889 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:34.145906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:34.147620 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:34.149880 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.149885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.149896 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.150320 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.150656 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.150659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.150661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.150864 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.151268 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:34.151272 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:34.151497 (+ 225us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.151660 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.151924 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260808 18:33:34.154075 30258 heartbeater.cc:347] Connected to a master server at 127.27.251.254:43293
I20260808 18:33:34.154289 30258 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:34.154839 30258 heartbeater.cc:511] Master 127.27.251.254:43293 requested a full tablet report, sending...
I20260808 18:33:34.155990 29875 ts_manager.cc:194] Registered new tserver with Master: bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194:38249)
I20260808 18:33:34.156718 29875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:36269
I20260808 18:33:34.159878 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30113
I20260808 18:33:34.159991 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:34.160324 28655 external_mini_cluster.cc:1428] Setting key 8b4112cd0a4b01f3d1ab0b18bcaad59d
I20260808 18:33:34.166733 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.156616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36647 (local address 127.27.251.254:43293)
0808 18:33:34.156754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.156758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.157224 (+ 466us) server_negotiation.cc:407] Connection header received
0808 18:33:34.157335 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.157337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.157376 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.157441 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.157986 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.158483 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.159379 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.159642 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.159783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.161208 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.161212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.161215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.161243 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.164869 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.165438 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.165441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.165442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.165521 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.165832 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.165835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.165836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.166051 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.166135 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.166514 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.166615 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:33:34.166895 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.156508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.194:36647)
0808 18:33:34.157140 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.157152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.157230 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.157481 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.157484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.157725 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:34.157844 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.157852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.158619 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.158622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.159252 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.159261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.159788 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.159791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.159900 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.160611 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:34.160623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:34.161074 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:34.164996 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.165001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.165003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.165298 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.165620 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.165626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.165630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.165739 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.166147 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:34.166153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:34.166304 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.166419 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.166617 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:34.192101 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:43293
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:34.308851 30270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:34.309161 30270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:34.309244 30270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:34.313251 30270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:34.313344 30270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:34.313426 30270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:34.313516 30270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:34.318933 30270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:34.320037 30270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:34.321220 30270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:34.330976 30275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:34.330993 30276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:34.330997 30278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:34.331384 30270 server_base.cc:1034] running on GCE node
I20260808 18:33:34.331815 30270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:34.332588 30270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:34.333786 30270 hybrid_clock.cc:648] HybridClock initialized: now 1786214014333737 us; error 60 us; skew 500 ppm
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:34.337008 30270 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:34.338044 30270 webserver.cc:460] Webserver started at http://127.27.251.195:45541/ using document root <none> and password file <none>
I20260808 18:33:34.338706 30270 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:34.338799 30270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:34.339049 30270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:34.341349 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "0e466d13668e4ceca8df6d99655e5457"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "6169efd31065747dee153af706437385"
server_key_iv: "a8147fa04c59d4290e3172af33d995f4"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.341974 30270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "0e466d13668e4ceca8df6d99655e5457"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "6169efd31065747dee153af706437385"
server_key_iv: "a8147fa04c59d4290e3172af33d995f4"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.346155 30270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:34.348958 30286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:34.350250 30270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:33:34.350389 30270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "0e466d13668e4ceca8df6d99655e5457"
format_stamp: "Formatted at 2026-08-08 18:33:34 on dist-test-slave-fd7l"
server_key: "6169efd31065747dee153af706437385"
server_key_iv: "a8147fa04c59d4290e3172af33d995f4"
server_key_version: "encryptionkey@0"
I20260808 18:33:34.350512 30270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:34.380613 30270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:34.384080 30270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:34.384285 30270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:34.385793 30270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:34.385869 30270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:34.385941 30270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:34.385982 30270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:34.397401 30270 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:35603
I20260808 18:33:34.397471 30414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:35603 every 8 connection(s)
I20260808 18:33:34.398475 30270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:34.399570 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30270
I20260808 18:33:34.399708 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:34.400007 28655 external_mini_cluster.cc:1428] Setting key 4b43c5f93a4f5e57c43f10dd2c6959af
I20260808 18:33:34.400396 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.386763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:47205 (local address 127.27.251.254:43293)
0808 18:33:34.386888 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.386892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.387973 (+ 1081us) server_negotiation.cc:407] Connection header received
0808 18:33:34.389230 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.389236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.389303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.389366 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.391140 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.391757 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.393264 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.393487 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.393598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.396350 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.396354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.396357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.396390 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.398009 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.398606 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.398612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.398614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.398706 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.399117 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.399122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.399124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.399392 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.399492 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.400086 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.400215 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:33:34.401398 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.387315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.195:47205)
0808 18:33:34.387821 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.387865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.388932 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.389811 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.389821 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.390255 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:34.390968 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.390992 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.391882 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.391886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.393098 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.393108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.393669 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.393673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.393831 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.394490 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:34.394515 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:34.396161 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:34.398129 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.398135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.398149 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.398493 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.398807 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.398812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.398815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.399007 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.399553 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:34.399558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:34.399828 (+ 270us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.399993 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.400307 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":156,"threads_started":1}
I20260808 18:33:34.402622 30415 heartbeater.cc:347] Connected to a master server at 127.27.251.254:43293
I20260808 18:33:34.402761 30415 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:34.403304 30415 heartbeater.cc:511] Master 127.27.251.254:43293 requested a full tablet report, sending...
I20260808 18:33:34.404439 29873 ts_manager.cc:194] Registered new tserver with Master: 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603)
I20260808 18:33:34.405262 29873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:47205
I20260808 18:33:34.411593 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.404977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:50019 (local address 127.27.251.254:43293)
0808 18:33:34.405085 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.405089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.405233 (+ 144us) server_negotiation.cc:407] Connection header received
0808 18:33:34.405307 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.405310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.405354 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.405434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.405953 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.406502 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.407309 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.407528 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.407648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.408749 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.408752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.408755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.408786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.409962 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.410382 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.410386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.410388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.410470 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.410755 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.410760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.410761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.410948 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.411076 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.411337 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.411445 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:33:34.411607 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.404882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.27.251.195:50019)
0808 18:33:34.405144 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.405160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.405244 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.405488 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.405491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.405736 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:34.405836 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.405842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.406629 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.406632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.407189 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.407198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.407649 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.407652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.407755 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.408249 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:34.408262 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:34.408619 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:34.410083 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.410088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.410090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.410282 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.410579 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.410581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.410583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.410666 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.411066 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:34.411069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:34.411172 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.411282 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.411422 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":105,"threads_started":1}
I20260808 18:33:34.414161 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:34.441839 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.433426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.27.251.254:43293)
0808 18:33:34.433568 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.433572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.433586 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:34.433635 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.433638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.433682 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.433767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.434544 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.435089 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.435917 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.436142 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.436236 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.438784 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.438788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.438790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.438813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.440165 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.440709 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.440712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.440713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.440793 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.441054 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.441056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.441057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.441227 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.441326 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.441620 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.441715 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:33:34.445142 29873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:34.448194 29873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:34.463251 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.456918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:35603 (local address 127.0.0.1:46270)
0808 18:33:34.457588 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.457603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.457677 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.458830 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.458852 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.458874 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.458966 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.458972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.460274 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.460280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.461430 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.461442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462584 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.462749 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.462826 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.462941 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.463088 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":89,"thread_start_us":109,"threads_started":1}
I20260808 18:33:34.463267 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.456985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.27.251.194:38249)
0808 18:33:34.457967 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.457979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.458001 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:33:34.458092 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.458097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.458306 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.458461 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.458904 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.459718 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.461453 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462420 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462577 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.462720 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.462992 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.463097 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":112,"threads_started":1}
I20260808 18:33:34.463444 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.456840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.27.251.193:42623)
0808 18:33:34.457967 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.457976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.458001 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:33:34.458091 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.458097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.458306 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.458461 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.458783 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.459963 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.461681 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462461 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462847 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.462973 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.463175 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.463268 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":77,"threads_started":1}
I20260808 18:33:34.463402 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.456688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42623 (local address 127.0.0.1:37872)
0808 18:33:34.457872 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.457887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.457973 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.458464 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.458470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.458494 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.458651 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.458669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.460097 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.460100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.461568 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.461577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462605 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.463037 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.463060 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.463128 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.463188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":847,"thread_start_us":106,"threads_started":1}
I20260808 18:33:34.464082 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.457491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.27.251.195:35603)
0808 18:33:34.458215 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.458224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.458242 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:33:34.458304 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.458308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.458502 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.458670 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.459106 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.460169 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.461568 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462220 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462370 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.462521 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.463822 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.463925 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":156,"threads_started":1}
I20260808 18:33:34.463637 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.456688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38249 (local address 127.0.0.1:47416)
0808 18:33:34.457480 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.457532 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.457677 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.458633 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.458635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.458651 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.458775 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.458781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.459841 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.459845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.461317 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.461328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.462553 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.462558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.462703 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.462826 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.462941 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.463515 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":569,"thread_start_us":88,"threads_started":1}
I20260808 18:33:34.466621 30027 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8bb0830cd4933a62824c950604424 (DEFAULT_TABLE table=test-table [id=ae4c1703a0ea4654b230de5df6d15f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.466799 30342 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8bb0830cd4933a62824c950604424 (DEFAULT_TABLE table=test-table [id=ae4c1703a0ea4654b230de5df6d15f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.467111 30187 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed8bb0830cd4933a62824c950604424 (DEFAULT_TABLE table=test-table [id=ae4c1703a0ea4654b230de5df6d15f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.467538 30027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8bb0830cd4933a62824c950604424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.467900 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8bb0830cd4933a62824c950604424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.467983 30187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed8bb0830cd4933a62824c950604424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.473699 30443 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Bootstrap starting.
I20260808 18:33:34.475555 30444 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Bootstrap starting.
I20260808 18:33:34.476054 30443 tablet_bootstrap.cc:654] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.476853 30445 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Bootstrap starting.
I20260808 18:33:34.476990 30443 log.cc:824] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:34.477754 30444 tablet_bootstrap.cc:654] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.478586 30444 log.cc:824] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:34.478852 30445 tablet_bootstrap.cc:654] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.479323 30443 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: No bootstrap required, opened a new log
I20260808 18:33:34.479521 30443 ts_tablet_manager.cc:1397] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:34.479557 30445 log.cc:824] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:34.481406 30445 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: No bootstrap required, opened a new log
I20260808 18:33:34.481628 30445 ts_tablet_manager.cc:1397] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:33:34.482131 30444 tablet_bootstrap.cc:492] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: No bootstrap required, opened a new log
I20260808 18:33:34.482349 30444 ts_tablet_manager.cc:1397] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:33:34.484416 30443 raft_consensus.cc:348] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.484710 30443 raft_consensus.cc:374] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.484767 30443 raft_consensus.cc:729] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6217e607afc41feaa98019c4bafcf80, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.485237 30443 consensus_queue.cc:260] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.485214 30445 raft_consensus.cc:348] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.485420 30445 raft_consensus.cc:374] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.485471 30445 raft_consensus.cc:729] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e466d13668e4ceca8df6d99655e5457, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.485879 30445 consensus_queue.cc:260] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.486203 30443 ts_tablet_manager.cc:1428] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260808 18:33:34.486199 30444 raft_consensus.cc:348] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.486433 30444 raft_consensus.cc:374] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.486488 30444 raft_consensus.cc:729] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5258f7bed54566b5d4ff63102f32a1, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.486582 30102 heartbeater.cc:503] Master 127.27.251.254:43293 was elected leader, sending a full tablet report...
I20260808 18:33:34.486897 30445 ts_tablet_manager.cc:1428] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:33:34.486860 30444 consensus_queue.cc:260] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.487432 30258 heartbeater.cc:503] Master 127.27.251.254:43293 was elected leader, sending a full tablet report...
I20260808 18:33:34.487654 30444 ts_tablet_manager.cc:1428] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:33:34.487900 30415 heartbeater.cc:503] Master 127.27.251.254:43293 was elected leader, sending a full tablet report...
I20260808 18:33:34.566352 30454 raft_consensus.cc:482] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:34.566593 30454 raft_consensus.cc:504] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.567828 30454 leader_election.cc:288] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603)
I20260808 18:33:34.572299 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.568267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:56537 (local address 127.27.251.195:35603)
0808 18:33:34.568392 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.568398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.568420 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:33:34.568544 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.568548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.568603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.568691 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.569099 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.569919 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.571160 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.571764 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.571858 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.571901 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.572138 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.572196 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:34.572330 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.568131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:35603 (local address 127.27.251.194:56537)
0808 18:33:34.568276 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.568293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.568401 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.568714 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.568717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.568736 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.568960 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.568968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.570026 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.570029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.571024 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.571033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.571879 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.571882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.572003 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.572029 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.572109 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.572184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":21248}
I20260808 18:33:34.573004 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.568136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42623 (local address 127.27.251.194:38841)
0808 18:33:34.568691 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.568706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.568826 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.569088 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.569090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.569108 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.569678 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.569689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.570602 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.570605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.571559 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.571570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.572569 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.572575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.572677 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.572715 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.572800 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.572862 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":139,"thread_start_us":133,"threads_started":1}
I20260808 18:33:34.573019 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.568184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:38841 (local address 127.27.251.193:42623)
0808 18:33:34.568345 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.568350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.568792 (+ 442us) server_negotiation.cc:407] Connection header received
0808 18:33:34.568931 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.568934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.568986 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.569082 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.569801 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.570489 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.571683 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.572306 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.572422 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.572481 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.572846 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.572926 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 18:33:34.573046 30362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8bb0830cd4933a62824c950604424" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e466d13668e4ceca8df6d99655e5457" is_pre_election: true
I20260808 18:33:34.573392 30362 raft_consensus.cc:2454] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 0.
I20260808 18:33:34.573652 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8bb0830cd4933a62824c950604424" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80" is_pre_election: true
I20260808 18:33:34.573975 30051 raft_consensus.cc:2454] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 0.
I20260808 18:33:34.573949 30135 leader_election.cc:302] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e466d13668e4ceca8df6d99655e5457, bf5258f7bed54566b5d4ff63102f32a1; no voters:
I20260808 18:33:34.574255 30454 raft_consensus.cc:2781] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:34.574366 30454 raft_consensus.cc:482] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:34.574431 30454 raft_consensus.cc:3037] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.575515 30454 raft_consensus.cc:504] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.575896 30454 leader_election.cc:288] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 election: Requested vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603)
I20260808 18:33:34.576238 30362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8bb0830cd4933a62824c950604424" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e466d13668e4ceca8df6d99655e5457"
I20260808 18:33:34.576263 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed8bb0830cd4933a62824c950604424" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80"
I20260808 18:33:34.576341 30362 raft_consensus.cc:3037] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.576392 30051 raft_consensus.cc:3037] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.577451 30362 raft_consensus.cc:2454] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 1.
I20260808 18:33:34.577452 30051 raft_consensus.cc:2454] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 1.
I20260808 18:33:34.577744 30135 leader_election.cc:302] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e466d13668e4ceca8df6d99655e5457, bf5258f7bed54566b5d4ff63102f32a1; no voters:
I20260808 18:33:34.577921 30454 raft_consensus.cc:2781] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:34.578202 30454 raft_consensus.cc:686] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 LEADER]: Becoming Leader. State: Replica: bf5258f7bed54566b5d4ff63102f32a1, State: Running, Role: LEADER
I20260808 18:33:34.578537 30454 consensus_queue.cc:237] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.581867 29873 catalog_manager.cc:5495] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:34.624540 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.620810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.27.251.194:38249)
0808 18:33:34.620963 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.620966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.620982 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:33:34.621222 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.621227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.621299 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.621384 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:34.621701 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.622250 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.623142 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.623363 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.623445 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.623725 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:34.624125 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:34.624209 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.624300 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.624349 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:33:34.630134 30051 raft_consensus.cc:1261] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Refusing update from remote peer bf5258f7bed54566b5d4ff63102f32a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:34.630538 30362 raft_consensus.cc:1261] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Refusing update from remote peer bf5258f7bed54566b5d4ff63102f32a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:34.630882 30458 consensus_queue.cc:1035] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:34.631289 30454 consensus_queue.cc:1035] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:34.639127 30461 mvcc.cc:205] Tried to move back new op lower bound from 7316332603917713408 to 7316332603716423680. Current Snapshot: MvccSnapshot[applied={T|T < 7316332603917713408}]
I20260808 18:33:34.639310 30463 mvcc.cc:205] Tried to move back new op lower bound from 7316332603917713408 to 7316332603716423680. Current Snapshot: MvccSnapshot[applied={T|T < 7316332603917713408 or (T in {7316332603917713408})}]
I20260808 18:33:34.649653 29874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:34.649904 29874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:34.652557 29874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Sending DeleteTablet for 3 replicas of tablet 4ed8bb0830cd4933a62824c950604424
I20260808 18:33:34.653457 30027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed8bb0830cd4933a62824c950604424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:34 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37872
I20260808 18:33:34.653527 30187 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed8bb0830cd4933a62824c950604424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:34 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:47416
I20260808 18:33:34.653975 30471 tablet_replica.cc:330] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: stopping tablet replica
I20260808 18:33:34.654055 30342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed8bb0830cd4933a62824c950604424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:34 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:46270
I20260808 18:33:34.654340 30471 raft_consensus.cc:2229] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:34.654335 29874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:33:34.654574 30473 tablet_replica.cc:330] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: stopping tablet replica
I20260808 18:33:34.654675 30471 raft_consensus.cc:2258] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:34.654989 30473 raft_consensus.cc:2229] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 18:33:34.655041 29874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:34.655304 30473 raft_consensus.cc:2258] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:34.656131 30471 ts_tablet_manager.cc:1905] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:34.658808 30472 tablet_replica.cc:330] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: stopping tablet replica
I20260808 18:33:34.659124 30472 raft_consensus.cc:2229] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:34.659165 30471 ts_tablet_manager.cc:1918] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:34.659247 30471 log.cc:1196] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/4ed8bb0830cd4933a62824c950604424
I20260808 18:33:34.659578 30471 ts_tablet_manager.cc:1939] T 4ed8bb0830cd4933a62824c950604424 P c6217e607afc41feaa98019c4bafcf80: Deleting consensus metadata
I20260808 18:33:34.660296 30187 tablet_service.cc:1467] Processing CreateTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 (DEFAULT_TABLE table=test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.660485 30027 tablet_service.cc:1467] Processing CreateTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 (DEFAULT_TABLE table=test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.660598 30187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ea7462d0804dc5a45cdc1add59f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.660753 30027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ea7462d0804dc5a45cdc1add59f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.662943 30443 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Bootstrap starting.
I20260808 18:33:34.663090 30444 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Bootstrap starting.
I20260808 18:33:34.663205 30342 tablet_service.cc:1467] Processing CreateTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 (DEFAULT_TABLE table=test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:34.663443 30473 ts_tablet_manager.cc:1905] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:34.663472 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ea7462d0804dc5a45cdc1add59f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.664019 30443 tablet_bootstrap.cc:654] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.664021 30444 tablet_bootstrap.cc:654] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.665479 30443 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: No bootstrap required, opened a new log
I20260808 18:33:34.665572 30443 ts_tablet_manager.cc:1397] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 18:33:34.665973 30473 ts_tablet_manager.cc:1918] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:34.665993 30443 raft_consensus.cc:348] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.666074 30473 log.cc:1196] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/4ed8bb0830cd4933a62824c950604424
I20260808 18:33:34.666108 30443 raft_consensus.cc:374] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.666141 30443 raft_consensus.cc:729] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6217e607afc41feaa98019c4bafcf80, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.666251 30443 consensus_queue.cc:260] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.666386 30473 ts_tablet_manager.cc:1939] T 4ed8bb0830cd4933a62824c950604424 P 0e466d13668e4ceca8df6d99655e5457: Deleting consensus metadata
I20260808 18:33:34.666479 30443 ts_tablet_manager.cc:1428] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:34.667225 30445 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Bootstrap starting.
I20260808 18:33:34.667670 29861 catalog_manager.cc:4841] TS 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603): tablet 4ed8bb0830cd4933a62824c950604424 (table test-table [id=ae4c1703a0ea4654b230de5df6d15f98]) successfully deleted
I20260808 18:33:34.667779 30472 raft_consensus.cc:2258] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:34.668154 30445 tablet_bootstrap.cc:654] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.669394 30472 ts_tablet_manager.cc:1905] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:34.669842 30445 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: No bootstrap required, opened a new log
I20260808 18:33:34.669914 30445 ts_tablet_manager.cc:1397] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:34.670185 29860 catalog_manager.cc:4841] TS c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623): tablet 4ed8bb0830cd4933a62824c950604424 (table test-table [id=ae4c1703a0ea4654b230de5df6d15f98]) successfully deleted
I20260808 18:33:34.670316 30445 raft_consensus.cc:348] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.670413 30445 raft_consensus.cc:374] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.670442 30445 raft_consensus.cc:729] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e466d13668e4ceca8df6d99655e5457, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.670569 30445 consensus_queue.cc:260] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.670795 30445 ts_tablet_manager.cc:1428] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:33:34.671824 30444 tablet_bootstrap.cc:492] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: No bootstrap required, opened a new log
I20260808 18:33:34.671918 30444 ts_tablet_manager.cc:1397] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260808 18:33:34.672170 30472 ts_tablet_manager.cc:1918] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:34.672266 30472 log.cc:1196] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/4ed8bb0830cd4933a62824c950604424
I20260808 18:33:34.672425 30444 raft_consensus.cc:348] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.672576 30444 raft_consensus.cc:374] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.672624 30472 ts_tablet_manager.cc:1939] T 4ed8bb0830cd4933a62824c950604424 P bf5258f7bed54566b5d4ff63102f32a1: Deleting consensus metadata
I20260808 18:33:34.672645 30444 raft_consensus.cc:729] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5258f7bed54566b5d4ff63102f32a1, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.672773 30444 consensus_queue.cc:260] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.673010 30444 ts_tablet_manager.cc:1428] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:34.673671 29859 catalog_manager.cc:4841] TS bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194:38249): tablet 4ed8bb0830cd4933a62824c950604424 (table test-table [id=ae4c1703a0ea4654b230de5df6d15f98]) successfully deleted
I20260808 18:33:34.814918 30453 raft_consensus.cc:482] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:34.815156 30453 raft_consensus.cc:504] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.816596 30453 leader_election.cc:288] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194:38249)
I20260808 18:33:34.820865 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.817060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:49317 (local address 127.27.251.194:38249)
0808 18:33:34.817191 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.817196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.817397 (+ 201us) server_negotiation.cc:407] Connection header received
0808 18:33:34.817562 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.817564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.817731 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.817825 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.818207 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.818738 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.819843 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.820237 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.820372 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.820428 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.820653 (+ 225us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.820726 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:33:34.820865 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.816765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42623 (local address 127.27.251.195:39825)
0808 18:33:34.817010 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.817029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.817151 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.817730 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.817733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.817756 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.817933 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.817942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.818781 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.818788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.819618 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.819627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.820350 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.820354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.820559 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.820582 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.820659 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.820725 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":4}
I20260808 18:33:34.820894 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.817298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:39825 (local address 127.27.251.193:42623)
0808 18:33:34.817439 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.817442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.817455 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:34.817501 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.817503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.817542 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.817617 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:34.818095 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.818649 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.819754 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.820221 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.820351 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.820414 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.820701 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.820788 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:33:34.820891 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.816821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38249 (local address 127.27.251.195:49317)
0808 18:33:34.817278 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.817298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.817437 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.817857 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.817859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.817877 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:34.818078 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.818087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.818876 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.818879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.819701 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.819713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.820402 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.820406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.820501 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.820538 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.820602 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.820690 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":48,"thread_start_us":130,"threads_started":1}
I20260808 18:33:34.821501 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ea7462d0804dc5a45cdc1add59f5c6" candidate_uuid: "0e466d13668e4ceca8df6d99655e5457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80" is_pre_election: true
I20260808 18:33:34.821604 30208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ea7462d0804dc5a45cdc1add59f5c6" candidate_uuid: "0e466d13668e4ceca8df6d99655e5457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5258f7bed54566b5d4ff63102f32a1" is_pre_election: true
I20260808 18:33:34.821676 30051 raft_consensus.cc:2454] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e466d13668e4ceca8df6d99655e5457 in term 0.
I20260808 18:33:34.821839 30208 raft_consensus.cc:2454] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e466d13668e4ceca8df6d99655e5457 in term 0.
I20260808 18:33:34.822173 30289 leader_election.cc:302] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e466d13668e4ceca8df6d99655e5457, c6217e607afc41feaa98019c4bafcf80; no voters:
I20260808 18:33:34.822480 30453 raft_consensus.cc:2781] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:34.822546 30453 raft_consensus.cc:482] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:34.822584 30453 raft_consensus.cc:3037] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.823756 30453 raft_consensus.cc:504] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.824088 30453 leader_election.cc:288] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [CANDIDATE]: Term 1 election: Requested vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194:38249)
I20260808 18:33:34.824446 30208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ea7462d0804dc5a45cdc1add59f5c6" candidate_uuid: "0e466d13668e4ceca8df6d99655e5457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf5258f7bed54566b5d4ff63102f32a1"
I20260808 18:33:34.824446 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ea7462d0804dc5a45cdc1add59f5c6" candidate_uuid: "0e466d13668e4ceca8df6d99655e5457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80"
I20260808 18:33:34.824635 30051 raft_consensus.cc:3037] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.824635 30208 raft_consensus.cc:3037] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.825817 30051 raft_consensus.cc:2454] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e466d13668e4ceca8df6d99655e5457 in term 1.
I20260808 18:33:34.825817 30208 raft_consensus.cc:2454] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e466d13668e4ceca8df6d99655e5457 in term 1.
I20260808 18:33:34.826115 30288 leader_election.cc:302] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e466d13668e4ceca8df6d99655e5457, bf5258f7bed54566b5d4ff63102f32a1; no voters:
I20260808 18:33:34.826287 30453 raft_consensus.cc:2781] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:34.826505 30453 raft_consensus.cc:686] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 1 LEADER]: Becoming Leader. State: Replica: 0e466d13668e4ceca8df6d99655e5457, State: Running, Role: LEADER
I20260808 18:33:34.826772 30453 consensus_queue.cc:237] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.829406 29874 catalog_manager.cc:5495] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "0e466d13668e4ceca8df6d99655e5457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } health_report { overall_health: HEALTHY } } }
Aug 08 18:33:34 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:34.882619 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.874399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43293 (local address 127.0.0.1:43910)
0808 18:33:34.874784 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.874798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.874910 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.875163 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.875165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.875582 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:34.875680 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.875685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.876410 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.876412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.877148 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.877158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.877584 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.877587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.877650 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.878180 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:34.878191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:34.879653 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:34.881235 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.881244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.881249 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.881478 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.881717 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:34.881719 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:34.881720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:34.881800 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:34.882154 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:34.882159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:34.882258 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.882350 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.882459 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":190,"threads_started":1}
I20260808 18:33:34.882894 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.874489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.27.251.254:43293)
0808 18:33:34.874658 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.874663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.874888 (+ 225us) server_negotiation.cc:407] Connection header received
0808 18:33:34.874993 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.874996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.875051 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.875123 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:34.875786 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.876314 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.877254 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.877476 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.877562 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.879784 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:34.879787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:34.879789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:34.879810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:34.881119 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.881564 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.881566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.881567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.881638 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:34.881875 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:34.881877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:34.881878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:34.882053 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:34.882127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.882409 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.882520 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 18:33:34.887099 29874 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:43910:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:33:34.887904 29874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:34.893040 30027 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef4bfd875bc4e7da133b232727e9518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842402ddc4d8458fa48c83535e96acf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:34.893168 30342 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef4bfd875bc4e7da133b232727e9518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842402ddc4d8458fa48c83535e96acf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:34.893486 30027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef4bfd875bc4e7da133b232727e9518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.893643 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef4bfd875bc4e7da133b232727e9518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.893054 30187 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef4bfd875bc4e7da133b232727e9518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842402ddc4d8458fa48c83535e96acf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:34.894867 30187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef4bfd875bc4e7da133b232727e9518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:34.896273 30445 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: Bootstrap starting.
I20260808 18:33:34.896308 30443 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: Bootstrap starting.
I20260808 18:33:34.897329 30445 tablet_bootstrap.cc:654] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.897629 30443 tablet_bootstrap.cc:654] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.898402 30444 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: Bootstrap starting.
I20260808 18:33:34.898986 30443 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: No bootstrap required, opened a new log
I20260808 18:33:34.899068 30443 ts_tablet_manager.cc:1397] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:34.899205 30444 tablet_bootstrap.cc:654] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:34.899247 30445 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: No bootstrap required, opened a new log
I20260808 18:33:34.899324 30445 ts_tablet_manager.cc:1397] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:34.899603 30443 raft_consensus.cc:348] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.899745 30443 raft_consensus.cc:374] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.899792 30443 raft_consensus.cc:729] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6217e607afc41feaa98019c4bafcf80, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.900147 30445 raft_consensus.cc:348] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.900290 30445 raft_consensus.cc:374] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.900339 30445 raft_consensus.cc:729] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e466d13668e4ceca8df6d99655e5457, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.900300 30443 consensus_queue.cc:260] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.900489 30445 consensus_queue.cc:260] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.900705 30452 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.900835 30452 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:34.900861 30445 ts_tablet_manager.cc:1428] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:34.900890 30444 tablet_bootstrap.cc:492] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: No bootstrap required, opened a new log
I20260808 18:33:34.900964 30444 ts_tablet_manager.cc:1397] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
W20260808 18:33:34.901089 30417 tablet.cc:2369] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:34.901124 30443 ts_tablet_manager.cc:1428] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 18:33:34.901320 30444 raft_consensus.cc:348] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.901413 30444 raft_consensus.cc:374] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:34.901438 30444 raft_consensus.cc:729] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5258f7bed54566b5d4ff63102f32a1, State: Initialized, Role: FOLLOWER
I20260808 18:33:34.901582 30444 consensus_queue.cc:260] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
W20260808 18:33:34.901757 30259 tablet.cc:2369] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:34.901806 30444 ts_tablet_manager.cc:1428] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:34.901823 30453 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.901834 30468 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.901953 30453 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:34.901968 30468 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 18:33:34.903179 30103 tablet.cc:2369] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:34.955013 30468 raft_consensus.cc:482] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:34.955196 30468 raft_consensus.cc:504] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.955596 30468 leader_election.cc:288] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603)
I20260808 18:33:34.956183 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef4bfd875bc4e7da133b232727e9518" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80" is_pre_election: true
I20260808 18:33:34.956190 30362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef4bfd875bc4e7da133b232727e9518" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e466d13668e4ceca8df6d99655e5457" is_pre_election: true
I20260808 18:33:34.956377 30051 raft_consensus.cc:2454] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 0.
I20260808 18:33:34.956377 30362 raft_consensus.cc:2454] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 0.
I20260808 18:33:34.956758 30134 leader_election.cc:302] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf5258f7bed54566b5d4ff63102f32a1, c6217e607afc41feaa98019c4bafcf80; no voters:
I20260808 18:33:34.956969 30468 raft_consensus.cc:2781] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:34.957034 30468 raft_consensus.cc:482] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:34.957075 30468 raft_consensus.cc:3037] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.958295 30468 raft_consensus.cc:504] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.958673 30468 leader_election.cc:288] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 election: Requested vote from peers c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623), 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603)
I20260808 18:33:34.959069 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef4bfd875bc4e7da133b232727e9518" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6217e607afc41feaa98019c4bafcf80"
I20260808 18:33:34.959107 30362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef4bfd875bc4e7da133b232727e9518" candidate_uuid: "bf5258f7bed54566b5d4ff63102f32a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e466d13668e4ceca8df6d99655e5457"
I20260808 18:33:34.959210 30051 raft_consensus.cc:3037] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.959221 30362 raft_consensus.cc:3037] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:34.960386 30362 raft_consensus.cc:2454] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 1.
I20260808 18:33:34.960386 30051 raft_consensus.cc:2454] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf5258f7bed54566b5d4ff63102f32a1 in term 1.
I20260808 18:33:34.960718 30134 leader_election.cc:302] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf5258f7bed54566b5d4ff63102f32a1, c6217e607afc41feaa98019c4bafcf80; no voters:
I20260808 18:33:34.960922 30468 raft_consensus.cc:2781] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:34.961040 30468 raft_consensus.cc:686] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 LEADER]: Becoming Leader. State: Replica: bf5258f7bed54566b5d4ff63102f32a1, State: Running, Role: LEADER
I20260808 18:33:34.961184 30468 consensus_queue.cc:237] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } }
I20260808 18:33:34.962029 30454 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: TxnStatusTablet state changed. Reason: New leader bf5258f7bed54566b5d4ff63102f32a1. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.962163 30454 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:34.962525 30493 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:33:34.962752 30493 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:33:34.963357 29872 catalog_manager.cc:5495] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:34.986529 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.982915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38249 (local address 127.0.0.1:47440)
0808 18:33:34.983069 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:34.983086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:34.983194 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:34.983481 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:34.983483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:34.983488 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:34.983577 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.983581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.984615 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.984618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.985222 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:34.985231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.985815 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.985818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:34.985891 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.985937 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:34.986239 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:34.986252 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 18:33:34.986306 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 18:33:34.986381 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 18:33:34.986527 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:34.983053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.27.251.194:38249)
0808 18:33:34.983182 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:34.983187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:34.983202 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:33:34.983315 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:34.983319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:34.983373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:34.983469 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:34.983671 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.984462 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.985337 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:34.985679 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:34.985784 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:34.986067 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:34.986149 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:34.986222 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:34.986342 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 18:33:34.986395 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:34.989310 30051 raft_consensus.cc:1261] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Refusing update from remote peer bf5258f7bed54566b5d4ff63102f32a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:34.989310 30362 raft_consensus.cc:1261] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Refusing update from remote peer bf5258f7bed54566b5d4ff63102f32a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:34.989760 30468 consensus_queue.cc:1035] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:34.990005 30454 consensus_queue.cc:1035] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:34.991248 30453 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: TxnStatusTablet state changed. Reason: New leader bf5258f7bed54566b5d4ff63102f32a1. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.991369 30453 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:34.991832 30452 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: TxnStatusTablet state changed. Reason: New leader bf5258f7bed54566b5d4ff63102f32a1. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.991966 30452 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:34.992916 30468 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.993018 30468 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:34.995352 30453 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.995461 30453 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P 0e466d13668e4ceca8df6d99655e5457: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:34.997484 30454 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.997587 30454 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P bf5258f7bed54566b5d4ff63102f32a1: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:34.999082 30452 tablet_replica.cc:439] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf5258f7bed54566b5d4ff63102f32a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 } } peers { permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 } } peers { permanent_uuid: "0e466d13668e4ceca8df6d99655e5457" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 35603 } } }
I20260808 18:33:34.999194 30452 tablet_replica.cc:442] T 6ef4bfd875bc4e7da133b232727e9518 P c6217e607afc41feaa98019c4bafcf80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:35.005044 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.000818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.27.251.195:35603)
0808 18:33:35.000936 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:35.000940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:35.001043 (+ 103us) server_negotiation.cc:407] Connection header received
0808 18:33:35.001210 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:35.001213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:35.001275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:35.001383 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:35.001602 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.002401 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.003311 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.003653 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.003747 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.004027 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:35.004651 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:35.004777 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:35.004836 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 18:33:35.004888 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 18:33:35.015722 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.012242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:52433 (local address 127.27.251.194:38249)
0808 18:33:35.012385 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:35.012390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:35.012404 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:35.012449 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:35.012452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:35.012527 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:35.012605 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:35.012875 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.013629 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.014545 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.014878 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.014987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.015245 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:35.015321 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:35.015391 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:35.015544 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 18:33:35.015596 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:33:35.015767 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.011710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38249 (local address 127.27.251.195:52433)
0808 18:33:35.012197 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:35.012213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:35.012315 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:35.012636 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:35.012639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:35.012646 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:35.012754 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:35.012760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.013770 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.013772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:35.014386 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:35.014395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.014994 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.014997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:35.015068 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.015111 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:35.015429 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:35.015436 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:33:35.015505 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 18:33:35.015568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":286,"threads_started":1}
I20260808 18:33:35.024339 30051 raft_consensus.cc:1261] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Refusing update from remote peer 0e466d13668e4ceca8df6d99655e5457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:35.024398 30208 raft_consensus.cc:1261] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Refusing update from remote peer 0e466d13668e4ceca8df6d99655e5457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:35.024855 30479 consensus_queue.cc:1035] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6217e607afc41feaa98019c4bafcf80" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:35.025087 30453 consensus_queue.cc:1035] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf5258f7bed54566b5d4ff63102f32a1" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:35.049976 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.045894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:49955 (local address 127.27.251.195:35603)
0808 18:33:35.046032 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:35.046037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:35.046250 (+ 213us) server_negotiation.cc:407] Connection header received
0808 18:33:35.046344 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:35.046347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:35.046396 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:35.046481 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:35.046880 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.047702 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.048960 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.049180 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.049263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.049582 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:35.049648 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:35.049703 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:35.049824 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 18:33:35.049870 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:33:35.050303 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.045825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:35603 (local address 127.27.251.194:49955)
0808 18:33:35.046167 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:35.046180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:35.046273 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:35.046564 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:35.046567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:35.046574 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:35.046757 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:35.046766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.047837 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.047840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:35.048839 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:35.048851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.049294 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.049298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:35.049388 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.049443 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:35.049735 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:35.049741 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:33:35.049807 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 18:33:35.049887 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":131,"threads_started":1}
I20260808 18:33:35.074469 29872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:35.074654 29872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:35.077039 29872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9eb5237ee4274cd7b094ce0b5b8f3477: Sending DeleteTablet for 3 replicas of tablet 16ea7462d0804dc5a45cdc1add59f5c6
I20260808 18:33:35.077554 30027 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:35 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37872
I20260808 18:33:35.077773 30471 tablet_replica.cc:330] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: stopping tablet replica
I20260808 18:33:35.077909 30471 raft_consensus.cc:2229] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:35.078037 30471 raft_consensus.cc:2258] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:35.078450 30342 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:35 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:46270
I20260808 18:33:35.078655 30473 tablet_replica.cc:330] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: stopping tablet replica
I20260808 18:33:35.078773 30471 ts_tablet_manager.cc:1905] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:35.078778 30473 raft_consensus.cc:2229] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:35.079049 30473 raft_consensus.cc:2258] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:35.079692 30473 ts_tablet_manager.cc:1905] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:35.081735 30471 ts_tablet_manager.cc:1918] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:35.081815 30471 log.cc:1196] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/16ea7462d0804dc5a45cdc1add59f5c6
I20260808 18:33:35.081957 30473 ts_tablet_manager.cc:1918] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:35.082021 30473 log.cc:1196] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/16ea7462d0804dc5a45cdc1add59f5c6
I20260808 18:33:35.082080 30471 ts_tablet_manager.cc:1939] T 16ea7462d0804dc5a45cdc1add59f5c6 P c6217e607afc41feaa98019c4bafcf80: Deleting consensus metadata
I20260808 18:33:35.082278 30473 ts_tablet_manager.cc:1939] T 16ea7462d0804dc5a45cdc1add59f5c6 P 0e466d13668e4ceca8df6d99655e5457: Deleting consensus metadata
I20260808 18:33:35.082772 29860 catalog_manager.cc:4841] TS c6217e607afc41feaa98019c4bafcf80 (127.27.251.193:42623): tablet 16ea7462d0804dc5a45cdc1add59f5c6 (table test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]) successfully deleted
I20260808 18:33:35.082875 29861 catalog_manager.cc:4841] TS 0e466d13668e4ceca8df6d99655e5457 (127.27.251.195:35603): tablet 16ea7462d0804dc5a45cdc1add59f5c6 (table test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]) successfully deleted
I20260808 18:33:35.082937 30187 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ea7462d0804dc5a45cdc1add59f5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:35 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:47416
I20260808 18:33:35.083148 30472 tablet_replica.cc:330] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: stopping tablet replica
I20260808 18:33:35.083285 30472 raft_consensus.cc:2229] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:35.083403 30472 raft_consensus.cc:2258] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:35.084148 30472 ts_tablet_manager.cc:1905] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:35.086874 30472 ts_tablet_manager.cc:1918] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:35.086943 30472 log.cc:1196] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/16ea7462d0804dc5a45cdc1add59f5c6
I20260808 18:33:35.087226 30472 ts_tablet_manager.cc:1939] T 16ea7462d0804dc5a45cdc1add59f5c6 P bf5258f7bed54566b5d4ff63102f32a1: Deleting consensus metadata
I20260808 18:33:35.087945 29859 catalog_manager.cc:4841] TS bf5258f7bed54566b5d4ff63102f32a1 (127.27.251.194:38249): tablet 16ea7462d0804dc5a45cdc1add59f5c6 (table test-table [id=e71f3f5f89544a92bfb037ba34d7bdee]) successfully deleted
Aug 08 18:33:35 dist-test-slave-fd7l krb5kdc[29823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:35.097922 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.084883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.27.251.193:42623)
0808 18:33:35.085013 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:35.085017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:35.085587 (+ 570us) server_negotiation.cc:407] Connection header received
0808 18:33:35.085778 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:35.085781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:35.085835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:35.085888 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:35.086893 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.087727 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.089040 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.089399 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.089510 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.093470 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:35.093484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:35.093502 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:35.093530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:35.095703 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:35.096227 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:35.096233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:35.096235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:35.096328 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:35.096668 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:35.096672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:35.096674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:35.096995 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:35.097221 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:35.097467 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 18:33:35.097727 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 18:33:35.098337 30094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37876
I20260808 18:33:35.113575 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:35.100205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.27.251.254:43293)
0808 18:33:35.100333 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:35.100338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:35.104806 (+ 4468us) server_negotiation.cc:407] Connection header received
0808 18:33:35.104945 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:35.104949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:35.105011 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:35.105092 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:35.105902 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.106724 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.107830 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:35.108191 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:35.108297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:35.110012 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:35.110016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:35.110019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:35.110048 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:35.111794 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:35.112271 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:35.112275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:35.112277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:35.112370 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:35.112680 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:35.112684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:35.112686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:35.112900 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:35.113037 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:35.113284 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 18:33:35.113407 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 18:33:35.113943 29872 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43922
I20260808 18:33:35.115684 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29954
I20260808 18:33:35.129231 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30113
I20260808 18:33:35.139568 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30270
I20260808 18:33:35.147013 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 29841
2026-08-08T18:33:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2966 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:35 dist-test-slave-fd7l krb5kdc[30513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:35 dist-test-slave-fd7l krb5kdc[30513](info): set up 2 sockets
Aug 08 18:33:35 dist-test-slave-fd7l krb5kdc[30513](info): commencing operation
krb5kdc: starting...
W20260808 18:33:36.247437 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:36 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:36.379076 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38795
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:36.495086 30529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:36.495404 30529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:36.495544 30529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:36.499635 30529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:36.499744 30529 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:33:36.499776 30529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:36.499821 30529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:36.499854 30529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:36.499909 30529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:36.505832 30529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38795
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:36.507064 30529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:36.508170 30529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:36.517402 30534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:36.517441 30535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:36.517588 30529 server_base.cc:1034] running on GCE node
W20260808 18:33:36.517434 30537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:36.518335 30529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:36.519526 30529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:36.520704 30529 hybrid_clock.cc:648] HybridClock initialized: now 1786214016520672 us; error 43 us; skew 500 ppm
Aug 08 18:33:36 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:36.523614 30529 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:36.524642 30529 webserver.cc:460] Webserver started at http://127.27.251.254:33021/ using document root <none> and password file <none>
I20260808 18:33:36.525257 30529 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:36.525352 30529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:36.525600 30529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:36.527819 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "7e6fdac017694f0e888bb0f02d78213f"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "5c25ee4dc97d9f25967b6a554b080a14"
server_key_iv: "90714b082d4e8a7546ba5075f5bcfce6"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.528414 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "7e6fdac017694f0e888bb0f02d78213f"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "5c25ee4dc97d9f25967b6a554b080a14"
server_key_iv: "90714b082d4e8a7546ba5075f5bcfce6"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.532253 30529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:33:36.534793 30544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:36.536149 30529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:36.536264 30529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "7e6fdac017694f0e888bb0f02d78213f"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "5c25ee4dc97d9f25967b6a554b080a14"
server_key_iv: "90714b082d4e8a7546ba5075f5bcfce6"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.536356 30529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:36.548072 30529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:36.551620 30529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:36.551785 30529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:36.560449 30529 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:38795
I20260808 18:33:36.560570 30612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:38795 every 8 connection(s)
I20260808 18:33:36.561545 30529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:36.564597 30613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:36.565863 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30529
I20260808 18:33:36.565964 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:36.566259 28655 external_mini_cluster.cc:1428] Setting key 760fc467e357b50fbc51407f6122203e
I20260808 18:33:36.571725 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Bootstrap starting.
I20260808 18:33:36.574549 30613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:36.575551 30613 log.cc:824] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:36 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:36.578146 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: No bootstrap required, opened a new log
I20260808 18:33:36.582509 30613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } }
I20260808 18:33:36.582795 30613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:36.582852 30613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6fdac017694f0e888bb0f02d78213f, State: Initialized, Role: FOLLOWER
I20260808 18:33:36.583213 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:36.567718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.27.251.254:38795)
0808 18:33:36.568178 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:36.568199 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:36.568245 (+ 46us) server_negotiation.cc:407] Connection header received
0808 18:33:36.569163 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:36.569187 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:36.569630 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:36.570016 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:36.570825 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.572249 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.573198 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.573888 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.574057 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:36.577214 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:36.577222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:36.577239 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:36.577270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:36.579950 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.580399 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.580405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.580412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.580596 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.580868 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.580871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.580873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.581395 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:36.581614 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:36.581820 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 18:33:36.582134 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20260808 18:33:36.583350 30613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } }
I20260808 18:33:36.583524 30613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:36.583595 30613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:36.583694 30613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:36.584863 30613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } }
I20260808 18:33:36.585233 30613 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6fdac017694f0e888bb0f02d78213f; no voters:
I20260808 18:33:36.585582 30613 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:36.585670 30618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:36.585912 30618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [term 1 LEADER]: Becoming Leader. State: Replica: 7e6fdac017694f0e888bb0f02d78213f, State: Running, Role: LEADER
I20260808 18:33:36.586252 30618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } }
I20260808 18:33:36.586972 30613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:36.588120 30619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6fdac017694f0e888bb0f02d78213f. Latest consensus state: current_term: 1 leader_uuid: "7e6fdac017694f0e888bb0f02d78213f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } } }
I20260808 18:33:36.588246 30619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:36.588346 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6fdac017694f0e888bb0f02d78213f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6fdac017694f0e888bb0f02d78213f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38795 } } }
I20260808 18:33:36.588435 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f [sys.catalog]: This master's current role is: LEADER
W20260808 18:33:36.592135 30636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:36.592206 30636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:36.592289 30628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:36.593503 30628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:36.599440 30628 catalog_manager.cc:1347] Generated new cluster ID: c21a05784a1e4c33a2a1fef950bf4379
I20260808 18:33:36.599516 30628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:36.612552 30628 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:36.613528 30628 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:36.619894 30628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Generated new TSK 0
I20260808 18:33:36.620633 30628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:36.656534 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:36.779234 30643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:36.779500 30643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:36.779546 30643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:36.783593 30643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:36.783650 30643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:36.783689 30643 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:33:36.783703 30643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:36.783789 30643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:36.789369 30643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:36.790398 30643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:36.791404 30643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:36.801144 30648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:36.801167 30649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:36.801247 30643 server_base.cc:1034] running on GCE node
W20260808 18:33:36.801144 30651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:36.801792 30643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:36.802424 30643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:36.803599 30643 hybrid_clock.cc:648] HybridClock initialized: now 1786214016803583 us; error 26 us; skew 500 ppm
Aug 08 18:33:36 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:36.806491 30643 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:36.807389 30643 webserver.cc:460] Webserver started at http://127.27.251.193:44851/ using document root <none> and password file <none>
I20260808 18:33:36.807972 30643 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:36.808033 30643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:36.808248 30643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:36.810324 30643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "e900636b10774432ae19cdda9c896a61"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "acd99a611c0d7fa719104968cb4b8e2d"
server_key_iv: "8cd02a9144e22a1f29f0c3c4f2bf656f"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.810876 30643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "e900636b10774432ae19cdda9c896a61"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "acd99a611c0d7fa719104968cb4b8e2d"
server_key_iv: "8cd02a9144e22a1f29f0c3c4f2bf656f"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.814668 30643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:36.817204 30658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:36.818434 30643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:36.818567 30643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "e900636b10774432ae19cdda9c896a61"
format_stamp: "Formatted at 2026-08-08 18:33:36 on dist-test-slave-fd7l"
server_key: "acd99a611c0d7fa719104968cb4b8e2d"
server_key_iv: "8cd02a9144e22a1f29f0c3c4f2bf656f"
server_key_version: "encryptionkey@0"
I20260808 18:33:36.818672 30643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:36.834816 30643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:36.837821 30643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:36.838032 30643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:36.839496 30643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:36.839551 30643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:36.839629 30643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:36.839671 30643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:36 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:36.851125 30643 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:40093
I20260808 18:33:36.851215 30786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:40093 every 8 connection(s)
I20260808 18:33:36.852098 30643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:36.854437 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30643
I20260808 18:33:36.854561 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:36.854895 28655 external_mini_cluster.cc:1428] Setting key 86f3b04b3627558d333a6342e161a407
I20260808 18:33:36.855157 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:36.840735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:37029 (local address 127.27.251.254:38795)
0808 18:33:36.840905 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:36.840910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:36.841937 (+ 1027us) server_negotiation.cc:407] Connection header received
0808 18:33:36.844033 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:36.844036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:36.844139 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:36.844256 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:36.846038 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.846753 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.848120 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.848369 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.848497 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:36.851149 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:36.851153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:36.851156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:36.851187 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:36.852964 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.853480 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.853483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.853485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.853581 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.853954 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.853957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.853959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.854150 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:36.854284 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:36.854859 (+ 575us) server_negotiation.cc:299] Negotiation successful
0808 18:33:36.854995 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 18:33:36.855938 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:36.841243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.193:37029)
0808 18:33:36.841763 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:36.841805 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:36.843774 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:36.844489 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:36.844500 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:36.845069 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:36.845848 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:36.845872 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.846896 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.846901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:36.847979 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:36.847989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.848553 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.848558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:36.848747 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:36.849386 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:36.849409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:36.850917 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:36.853087 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:36.853094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:36.853107 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:36.853367 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:36.853676 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:36.853680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:36.853682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:36.853839 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:36.854292 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:36.854297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:36.854562 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 18:33:36.854757 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 18:33:36.855069 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260808 18:33:36.857266 30787 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38795
I20260808 18:33:36.857419 30787 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:36.857908 30787 heartbeater.cc:511] Master 127.27.251.254:38795 requested a full tablet report, sending...
I20260808 18:33:36.859508 30564 ts_manager.cc:194] Registered new tserver with Master: e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093)
I20260808 18:33:36.860975 30564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:37029
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:36.868126 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:36.860013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:57617 (local address 127.27.251.254:38795)
0808 18:33:36.860142 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:36.860147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:36.860657 (+ 510us) server_negotiation.cc:407] Connection header received
0808 18:33:36.860799 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:36.860802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:36.860846 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:36.860943 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:36.861420 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.862010 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.862998 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.863221 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.863280 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:36.864850 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:36.864853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:36.864855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:36.864878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:36.866317 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.866840 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.866843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.866844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.866921 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:36.867279 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:36.867281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:36.867283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:36.867449 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:36.867536 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:36.867844 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 18:33:36.867996 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:33:36.868271 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:36.860242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.193:57617)
0808 18:33:36.860551 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:36.860567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:36.860697 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:36.860960 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:36.860963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:36.861194 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:36.861298 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:36.861304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.862156 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.862159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:36.862818 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:36.862827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:36.863421 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:36.863424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:36.863492 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:36.864095 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:36.864117 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:36.864698 (+ 581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:36.866441 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:36.866447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:36.866452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:36.866734 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:36.867041 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:36.867046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:36.867049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:36.867167 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:36.867567 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:36.867571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:36.867670 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 18:33:36.867786 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 18:33:36.867998 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:36.888055 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:37.003520 30798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:37.003782 30798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:37.003867 30798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:37.007961 30798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:37.008013 30798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:37.008051 30798 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:33:37.008067 30798 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:37.008100 30798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:37.013312 30798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:37.014370 30798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:37.015328 30798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:37.025110 30808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:37.025112 30803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:37.025236 30804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:37.025993 30798 server_base.cc:1034] running on GCE node
I20260808 18:33:37.026504 30798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:37.027305 30798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:37.028486 30798 hybrid_clock.cc:648] HybridClock initialized: now 1786214017028464 us; error 33 us; skew 500 ppm
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:37.031507 30798 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:37.032435 30798 webserver.cc:460] Webserver started at http://127.27.251.194:37849/ using document root <none> and password file <none>
I20260808 18:33:37.033061 30798 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:37.033133 30798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:37.033391 30798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:37.035490 30798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "9197a68efa6c1615cc828a7c65eb962c"
server_key_iv: "3fc1c70df1187b28a5498c8b27d69a02"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.036109 30798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "9197a68efa6c1615cc828a7c65eb962c"
server_key_iv: "3fc1c70df1187b28a5498c8b27d69a02"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.040092 30798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:37.042649 30816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:37.043954 30798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:37.044097 30798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "9197a68efa6c1615cc828a7c65eb962c"
server_key_iv: "3fc1c70df1187b28a5498c8b27d69a02"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.044214 30798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:37.062067 30798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:37.065241 30798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:37.065469 30798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:37.067090 30798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:37.067142 30798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:37.067219 30798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:37.067257 30798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:37.080025 30798 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:44283
I20260808 18:33:37.080067 30944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:44283 every 8 connection(s)
I20260808 18:33:37.081094 30798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:37.082731 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.067967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:46287 (local address 127.27.251.254:38795)
0808 18:33:37.068140 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.068145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.069091 (+ 946us) server_negotiation.cc:407] Connection header received
0808 18:33:37.070478 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.070485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.070560 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.070635 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.072717 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.073323 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.074510 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.074749 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.074878 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.078592 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.078596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.078599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.078632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.080379 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.080961 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.080965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.080967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.081058 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.081442 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.081445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.081447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.081694 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.081792 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.082477 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.082603 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 18:33:37.083523 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.068322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.194:46287)
0808 18:33:37.068891 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.068955 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.070176 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.071145 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.071158 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.071741 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:37.072542 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.072566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.073464 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.073467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.074361 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.074371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.074942 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.074947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.075169 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.075788 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:37.075818 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:37.078415 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:37.080535 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.080544 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.080572 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.080845 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.081160 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.081163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.081165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.081325 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.081847 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:37.081853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:37.082177 (+ 324us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.082403 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.082675 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":176,"threads_started":1}
I20260808 18:33:37.084986 30945 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38795
I20260808 18:33:37.085186 30945 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:37.085294 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30798
I20260808 18:33:37.085414 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:37.085772 28655 external_mini_cluster.cc:1428] Setting key bbbd8ca4d0463c3fe6a8a0564fc1bc06
I20260808 18:33:37.085793 30945 heartbeater.cc:511] Master 127.27.251.254:38795 requested a full tablet report, sending...
I20260808 18:33:37.087046 30564 ts_manager.cc:194] Registered new tserver with Master: dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283)
I20260808 18:33:37.087824 30564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:46287
I20260808 18:33:37.095957 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.087492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:43005 (local address 127.27.251.254:38795)
0808 18:33:37.087607 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.087614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.088816 (+ 1202us) server_negotiation.cc:407] Connection header received
0808 18:33:37.088932 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.088938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.088995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.089133 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.089678 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.090342 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.091171 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.091408 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.091516 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.092748 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.092751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.092754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.092778 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.094250 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.094681 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.094685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.094687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.094765 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.095072 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.095076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.095077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.095281 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.095395 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.095663 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.095777 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:33:37.095971 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.087387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.194:43005)
0808 18:33:37.088719 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.088735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.088839 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.089197 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.089200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.089424 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:37.089550 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.089570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.090474 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.090477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.091049 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.091057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.091524 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.091528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.091635 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.092174 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:37.092197 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:37.092604 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:37.094363 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.094368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.094370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.094580 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.094878 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.094881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.094884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.094973 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.095381 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:37.095384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:37.095488 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.095607 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.095761 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:37.115680 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:37.238058 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:37.238312 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:37.238359 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:37.242576 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:37.242638 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:37.242702 30956 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 18:33:37.242733 30956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:33:37.242794 30956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:37.248253 30956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:38795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:37.249393 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:37.250450 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:37.260703 30964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:37.260703 30961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:37.260759 30962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:37.261111 30956 server_base.cc:1034] running on GCE node
I20260808 18:33:37.261471 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:37.262193 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:37.263418 30956 hybrid_clock.cc:648] HybridClock initialized: now 1786214017263399 us; error 57 us; skew 500 ppm
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:37.266217 30956 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:37.267155 30956 webserver.cc:460] Webserver started at http://127.27.251.195:35675/ using document root <none> and password file <none>
I20260808 18:33:37.267727 30956 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:37.267822 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:37.268064 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:37.270208 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "02404116a27d4c86a579a2785e8d4b3b"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "14763b7c463ca1d0f0654ed77ea1ceaf"
server_key_iv: "35e4e7b66cbc00eeee590f4d8a6d44ad"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.270794 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "02404116a27d4c86a579a2785e8d4b3b"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "14763b7c463ca1d0f0654ed77ea1ceaf"
server_key_iv: "35e4e7b66cbc00eeee590f4d8a6d44ad"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.274665 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 18:33:37.277189 30972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:37.278482 30956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:37.278626 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "02404116a27d4c86a579a2785e8d4b3b"
format_stamp: "Formatted at 2026-08-08 18:33:37 on dist-test-slave-fd7l"
server_key: "14763b7c463ca1d0f0654ed77ea1ceaf"
server_key_iv: "35e4e7b66cbc00eeee590f4d8a6d44ad"
server_key_version: "encryptionkey@0"
I20260808 18:33:37.278746 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:37.319485 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:37.322686 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:37.322887 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:37.324497 30956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:37.324636 30956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:37.324720 30956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:37.324754 30956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:37.336791 30956 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:46843
I20260808 18:33:37.336854 31103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:46843 every 8 connection(s)
I20260808 18:33:37.337896 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:37.339772 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.326099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.195:36809)
0808 18:33:37.326593 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.326636 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.327484 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.328204 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.328214 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.328653 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:37.329190 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.329204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.330407 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.330409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.331502 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.331512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.332225 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.332231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.332446 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.333115 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:37.333131 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:37.334626 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:37.336831 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.336837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.336852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.337205 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.337496 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.337498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.337499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.337649 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.338153 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:37.338158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:37.338378 (+ 220us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.338600 (+ 222us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.338908 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260808 18:33:37.340353 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.325477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:36809 (local address 127.27.251.254:38795)
0808 18:33:37.325640 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.325644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.326745 (+ 1101us) server_negotiation.cc:407] Connection header received
0808 18:33:37.327812 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.327816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.327914 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.328000 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.329376 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.330290 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.331643 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.332040 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.332164 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.334787 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.334793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.334798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.334833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.336694 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.337303 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.337307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.337308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.337399 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.337744 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.337748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.337749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.338043 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.338150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.340146 (+ 1996us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.340246 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:33:37.341342 31104 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38795
I20260808 18:33:37.341504 31104 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:37.342036 31104 heartbeater.cc:511] Master 127.27.251.254:38795 requested a full tablet report, sending...
I20260808 18:33:37.343104 30561 ts_manager.cc:194] Registered new tserver with Master: 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.343640 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 30956
I20260808 18:33:37.343758 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:37.343887 30561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:36809
I20260808 18:33:37.344095 28655 external_mini_cluster.cc:1428] Setting key 3e5c11566c168bfada4f64fd548be485
I20260808 18:33:37.346140 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:33:37.351818 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.343516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:35231 (local address 127.27.251.254:38795)
0808 18:33:37.343634 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.343640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.344677 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 18:33:37.344816 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.344818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.344861 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.344933 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.345479 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.346190 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.346984 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.347293 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.347389 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.348617 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.348623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.348629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.348662 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.350121 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.350624 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.350627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.350628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.350699 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.350994 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.350996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.350997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.351150 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.351223 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.351608 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.351716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:37.351954 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.343613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.27.251.195:35231)
0808 18:33:37.344586 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.344600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.344726 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.345042 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.345045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.345249 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:37.345367 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.345375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.346306 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.346310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.346866 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.346873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.347394 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.347398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.347495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.347985 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:37.347996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:37.348402 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:37.350232 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.350238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.350243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.350512 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.350786 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.350792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.350796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.350900 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.351267 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:37.351273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:37.351389 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.351513 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.351689 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":85,"threads_started":1}
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:37.373634 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.365438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.27.251.254:38795)
0808 18:33:37.365560 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.365581 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.365594 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:37.365631 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.365633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.365671 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.365738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.366385 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.366905 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.367699 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.367901 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.367975 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.370578 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.370580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.370583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.370605 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.372100 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.372590 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.372593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.372594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.372670 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.372900 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.372902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.372903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.373083 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.373181 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.373374 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.373476 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:37.376456 30561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:37.379678 30561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:37.395473 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.389174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46843 (local address 127.0.0.1:58116)
0808 18:33:37.390412 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.390424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.390522 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.391124 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.391130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.391154 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:37.391300 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.391308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392965 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.392969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.394046 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.394057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.394871 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.394875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.395026 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.395076 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.395187 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.395309 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"mutex_wait_us":89,"thread_start_us":111,"threads_started":1}
I20260808 18:33:37.395521 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.389051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:44283 (local address 127.0.0.1:57410)
0808 18:33:37.390025 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.390090 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.390225 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.391424 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.391427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.391443 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:37.391576 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.391581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392965 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.392969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.394006 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.394017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.395152 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.395159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.395287 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.395308 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.395363 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.395415 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":31,"thread_start_us":73,"threads_started":1}
I20260808 18:33:37.395932 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.389880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.27.251.195:46843)
0808 18:33:37.390723 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.390734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.390754 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:33:37.390815 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.390819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.390961 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.391126 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:37.391418 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392847 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.394209 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.394740 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.394903 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.395054 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.395676 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.395781 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":123,"threads_started":1}
I20260808 18:33:37.396108 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.389274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.27.251.194:44283)
0808 18:33:37.389617 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.389629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.390663 (+ 1034us) server_negotiation.cc:407] Connection header received
0808 18:33:37.390750 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.390756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.390962 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.391126 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:37.391698 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392835 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.394151 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.395036 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.395197 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.395341 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.395429 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.395526 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260808 18:33:37.397116 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.390190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.0.0.1:37428)
0808 18:33:37.390458 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.390471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.390558 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.391341 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.391343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.391383 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:37.391536 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.391544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392858 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.392861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.395255 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.395267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.396610 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.396614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.396754 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.396786 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.396882 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.396967 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":66,"thread_start_us":99,"threads_started":1}
I20260808 18:33:37.397189 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.390629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.27.251.193:40093)
0808 18:33:37.390905 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.390912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.390934 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:33:37.391009 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.391013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.391211 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.391357 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:37.391667 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.392702 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.395411 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.396254 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.396395 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.396496 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.396933 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.397039 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":108,"threads_started":1}
I20260808 18:33:37.398748 31030 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 (DEFAULT_TABLE table=test-table [id=cf455183d3984e45bdad1b72f3d71c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.398902 30871 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 (DEFAULT_TABLE table=test-table [id=cf455183d3984e45bdad1b72f3d71c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.399731 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed1ce9860c45abb465a4e341b1b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.399744 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed1ce9860c45abb465a4e341b1b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.403843 30714 tablet_service.cc:1467] Processing CreateTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 (DEFAULT_TABLE table=test-table [id=cf455183d3984e45bdad1b72f3d71c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.405033 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ed1ce9860c45abb465a4e341b1b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.406760 31128 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Bootstrap starting.
I20260808 18:33:37.408897 31129 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Bootstrap starting.
I20260808 18:33:37.409157 31128 tablet_bootstrap.cc:654] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.410027 31128 log.cc:824] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:37.411142 31129 tablet_bootstrap.cc:654] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.411798 31128 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: No bootstrap required, opened a new log
I20260808 18:33:37.412002 31128 ts_tablet_manager.cc:1397] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:33:37.412011 31129 log.cc:824] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:37.412427 31131 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Bootstrap starting.
I20260808 18:33:37.414232 31129 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: No bootstrap required, opened a new log
I20260808 18:33:37.414469 31129 ts_tablet_manager.cc:1397] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:33:37.414599 31131 tablet_bootstrap.cc:654] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.415843 31131 log.cc:824] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:37.416081 31128 raft_consensus.cc:348] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.416378 31128 raft_consensus.cc:374] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.416446 31128 raft_consensus.cc:729] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd1bd16a9d64891833e04b8839f3c4c, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.416945 31128 consensus_queue.cc:260] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.417670 30945 heartbeater.cc:503] Master 127.27.251.254:38795 was elected leader, sending a full tablet report...
I20260808 18:33:37.417843 31131 tablet_bootstrap.cc:492] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: No bootstrap required, opened a new log
I20260808 18:33:37.417956 31128 ts_tablet_manager.cc:1428] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:33:37.418015 31131 ts_tablet_manager.cc:1397] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:33:37.418324 31129 raft_consensus.cc:348] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.418524 31129 raft_consensus.cc:374] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.418566 31129 raft_consensus.cc:729] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.418948 31129 consensus_queue.cc:260] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.419809 31129 ts_tablet_manager.cc:1428] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:33:37.420722 31104 heartbeater.cc:503] Master 127.27.251.254:38795 was elected leader, sending a full tablet report...
I20260808 18:33:37.422766 31131 raft_consensus.cc:348] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.423055 31131 raft_consensus.cc:374] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.423130 31131 raft_consensus.cc:729] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e900636b10774432ae19cdda9c896a61, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.423554 31131 consensus_queue.cc:260] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.424186 30787 heartbeater.cc:503] Master 127.27.251.254:38795 was elected leader, sending a full tablet report...
I20260808 18:33:37.424332 31131 ts_tablet_manager.cc:1428] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260808 18:33:37.472939 31137 raft_consensus.cc:482] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:37.473233 31137 raft_consensus.cc:504] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.474473 31137 leader_election.cc:288] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.478868 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.474861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:54383 (local address 127.27.251.195:46843)
0808 18:33:37.475013 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.475019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.475388 (+ 369us) server_negotiation.cc:407] Connection header received
0808 18:33:37.475558 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.475561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.475614 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.475700 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:37.476012 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.476631 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.477846 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.478257 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.478400 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.478455 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.478664 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.478740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:33:37.478896 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.474725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46843 (local address 127.27.251.193:54383)
0808 18:33:37.475288 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.475300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.475439 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.475714 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.475717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.475733 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:37.475894 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.475900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.476754 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.476758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.477720 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.477730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.478399 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.478403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.478506 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.478534 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.478647 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.478726 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":82,"thread_start_us":131,"threads_started":1}
I20260808 18:33:37.478889 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.474923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:55031 (local address 127.27.251.194:44283)
0808 18:33:37.475057 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.475062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.475462 (+ 400us) server_negotiation.cc:407] Connection header received
0808 18:33:37.475618 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.475621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.475671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.475754 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:37.476038 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.476644 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.477846 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.478258 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.478398 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.478457 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.478684 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.478757 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:33:37.478914 31138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.474725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:44283 (local address 127.27.251.193:55031)
0808 18:33:37.475361 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.475372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.475506 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.475765 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.475768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.475784 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:37.475927 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.475932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.476754 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.476757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.477714 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.477723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.478403 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.478407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.478545 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.478577 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.478668 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.478742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":322,"thread_start_us":119,"threads_started":1}
I20260808 18:33:37.479682 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed1ce9860c45abb465a4e341b1b060" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b" is_pre_election: true
I20260808 18:33:37.479935 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed1ce9860c45abb465a4e341b1b060" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" is_pre_election: true
I20260808 18:33:37.480036 31052 raft_consensus.cc:2454] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.480221 30893 raft_consensus.cc:2454] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.480589 30659 leader_election.cc:302] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.480904 31137 raft_consensus.cc:2781] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:37.481001 31137 raft_consensus.cc:482] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:37.481047 31137 raft_consensus.cc:3037] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.482179 31137 raft_consensus.cc:504] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.482538 31137 leader_election.cc:288] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Requested vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.483156 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed1ce9860c45abb465a4e341b1b060" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b"
I20260808 18:33:37.483162 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ed1ce9860c45abb465a4e341b1b060" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
I20260808 18:33:37.483270 31052 raft_consensus.cc:3037] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.483317 30893 raft_consensus.cc:3037] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.484355 31052 raft_consensus.cc:2454] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.484723 30659 leader_election.cc:302] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.484747 30893 raft_consensus.cc:2454] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.484935 31137 raft_consensus.cc:2781] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:37.485219 31137 raft_consensus.cc:686] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Becoming Leader. State: Replica: e900636b10774432ae19cdda9c896a61, State: Running, Role: LEADER
I20260808 18:33:37.485675 31137 consensus_queue.cc:237] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } }
I20260808 18:33:37.489677 30564 catalog_manager.cc:5495] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e900636b10774432ae19cdda9c896a61 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } health_report { overall_health: HEALTHY } } }
Aug 08 18:33:37 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:37.547353 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.538276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38795 (local address 127.0.0.1:58302)
0808 18:33:37.538595 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.538609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.538717 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.538964 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.538966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.539514 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:37.539650 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.539656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.540429 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.540433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.541112 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.541122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.541648 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.541651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.541738 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.542381 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:37.542394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:37.543918 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:37.545704 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.545713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.545718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.545978 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.546251 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:37.546254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:37.546255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:37.546345 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:37.546753 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:37.546759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:37.546888 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.547006 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.547193 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":133,"threads_started":1}
I20260808 18:33:37.547432 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.538373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.27.251.254:38795)
0808 18:33:37.538579 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.538584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.538689 (+ 105us) server_negotiation.cc:407] Connection header received
0808 18:33:37.538800 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.538802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.538846 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.538917 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:37.539762 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.540305 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.541236 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.541471 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.541646 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.544049 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:37.544052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:37.544054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:37.544077 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:37.545589 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.546071 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.546073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.546074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.546148 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:37.546431 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:37.546433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:37.546434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:37.546628 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:37.546706 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.547064 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.547291 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":128}
I20260808 18:33:37.551630 30564 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:58302:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:33:37.552423 30564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:37.557591 30714 tablet_service.cc:1467] Processing CreateTablet for tablet eed73ec551ce486b968db5890b719ef7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02147ecb0e3d4a6199a39fcad5b2e129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:37.557823 31030 tablet_service.cc:1467] Processing CreateTablet for tablet eed73ec551ce486b968db5890b719ef7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02147ecb0e3d4a6199a39fcad5b2e129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:37.557727 30871 tablet_service.cc:1467] Processing CreateTablet for tablet eed73ec551ce486b968db5890b719ef7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=02147ecb0e3d4a6199a39fcad5b2e129]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:33:37.558494 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed73ec551ce486b968db5890b719ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.558494 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed73ec551ce486b968db5890b719ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.558506 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed73ec551ce486b968db5890b719ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.561646 31128 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: Bootstrap starting.
I20260808 18:33:37.561692 31129 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: Bootstrap starting.
I20260808 18:33:37.562726 31129 tablet_bootstrap.cc:654] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.562726 31128 tablet_bootstrap.cc:654] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.564289 31129 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: No bootstrap required, opened a new log
I20260808 18:33:37.564370 31129 ts_tablet_manager.cc:1397] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:37.564639 31128 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: No bootstrap required, opened a new log
I20260808 18:33:37.564719 31128 ts_tablet_manager.cc:1397] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:37.564874 31129 raft_consensus.cc:348] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.564994 31129 raft_consensus.cc:374] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.565022 31129 raft_consensus.cc:729] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.565099 31131 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: Bootstrap starting.
I20260808 18:33:37.565127 31129 consensus_queue.cc:260] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.565183 31128 raft_consensus.cc:348] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.565281 31128 raft_consensus.cc:374] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.565311 31128 raft_consensus.cc:729] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd1bd16a9d64891833e04b8839f3c4c, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.565382 31129 ts_tablet_manager.cc:1428] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:37.565349 31136 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.565479 31136 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.565765 31128 consensus_queue.cc:260] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.566165 31131 tablet_bootstrap.cc:654] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.566637 31128 ts_tablet_manager.cc:1428] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:37.566601 31135 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.567366 31135 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.569289 31131 tablet_bootstrap.cc:492] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: No bootstrap required, opened a new log
I20260808 18:33:37.569412 31131 ts_tablet_manager.cc:1397] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 18:33:37.569823 31131 raft_consensus.cc:348] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.569941 31131 raft_consensus.cc:374] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.570012 31131 raft_consensus.cc:729] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e900636b10774432ae19cdda9c896a61, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.570170 31131 consensus_queue.cc:260] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.570451 31131 ts_tablet_manager.cc:1428] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:33:37.570546 31137 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.570689 31137 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 18:33:37.582348 30946 tablet.cc:2369] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:33:37.589054 31105 tablet.cc:2369] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:33:37.603586 30788 tablet.cc:2369] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:37.762140 31137 raft_consensus.cc:482] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:37.762341 31137 raft_consensus.cc:504] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.762845 31137 leader_election.cc:288] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.763499 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed73ec551ce486b968db5890b719ef7" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" is_pre_election: true
I20260808 18:33:37.763528 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed73ec551ce486b968db5890b719ef7" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b" is_pre_election: true
I20260808 18:33:37.763685 30893 raft_consensus.cc:2454] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.763691 31052 raft_consensus.cc:2454] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.764048 30659 leader_election.cc:302] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, dcd1bd16a9d64891833e04b8839f3c4c, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.764241 31137 raft_consensus.cc:2781] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:37.764329 31137 raft_consensus.cc:482] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:37.764381 31137 raft_consensus.cc:3037] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.765496 31137 raft_consensus.cc:504] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.765861 31137 leader_election.cc:288] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Requested vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.766271 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed73ec551ce486b968db5890b719ef7" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b"
I20260808 18:33:37.766271 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed73ec551ce486b968db5890b719ef7" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
I20260808 18:33:37.766404 30893 raft_consensus.cc:3037] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.766404 31052 raft_consensus.cc:3037] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.767649 31052 raft_consensus.cc:2454] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.767649 30893 raft_consensus.cc:2454] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.767979 30662 leader_election.cc:302] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcd1bd16a9d64891833e04b8839f3c4c, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.768182 31137 raft_consensus.cc:2781] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:37.768301 31137 raft_consensus.cc:686] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Becoming Leader. State: Replica: e900636b10774432ae19cdda9c896a61, State: Running, Role: LEADER
I20260808 18:33:37.768447 31137 consensus_queue.cc:237] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.769284 31140 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: TxnStatusTablet state changed. Reason: New leader e900636b10774432ae19cdda9c896a61. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.769409 31140 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:37.769801 31154 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:33:37.770047 31154 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:33:37.770499 30561 catalog_manager.cc:5495] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e900636b10774432ae19cdda9c896a61 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:37.804280 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.800013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.0.0.1:37440)
0808 18:33:37.800221 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.800237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.800328 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.800716 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.800718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.800725 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:37.800821 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.800826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.801874 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.801876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.802510 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.802519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.803119 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.803122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.803189 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.803234 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:37.804011 (+ 777us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:37.804024 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.804089 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.804158 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 18:33:37.804314 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.800137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.27.251.193:40093)
0808 18:33:37.800276 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.800282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.800300 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:33:37.800496 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.800499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.800593 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.800705 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:37.800935 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.801762 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.802634 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.802989 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.803099 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.803379 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:37.803916 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:37.804034 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.804108 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.804168 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:37.809906 31052 raft_consensus.cc:1261] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:37.809957 30893 raft_consensus.cc:1261] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:37.810671 31137 consensus_queue.cc:1035] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:37.810873 31140 consensus_queue.cc:1035] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:37.814400 31136 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: TxnStatusTablet state changed. Reason: New leader e900636b10774432ae19cdda9c896a61. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.814553 31136 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.814930 31135 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: TxnStatusTablet state changed. Reason: New leader e900636b10774432ae19cdda9c896a61. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.815056 31135 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.817903 31137 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.818034 31137 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:37.818157 31157 mvcc.cc:205] Tried to move back new op lower bound from 7316332616942166016 to 7316332616781275136. Current Snapshot: MvccSnapshot[applied={T|T < 7316332616942166016}]
I20260808 18:33:37.818197 31135 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.818295 31135 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P dcd1bd16a9d64891833e04b8839f3c4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.819888 31158 mvcc.cc:205] Tried to move back new op lower bound from 7316332616942166016 to 7316332616781275136. Current Snapshot: MvccSnapshot[applied={T|T < 7316332616942166016}]
I20260808 18:33:37.820338 31140 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.820449 31140 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P e900636b10774432ae19cdda9c896a61: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:33:37.821911 31136 tablet_replica.cc:439] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } } }
I20260808 18:33:37.822026 31136 tablet_replica.cc:442] T eed73ec551ce486b968db5890b719ef7 P 02404116a27d4c86a579a2785e8d4b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:33:37.823629 31159 mvcc.cc:205] Tried to move back new op lower bound from 7316332616942166016 to 7316332616781275136. Current Snapshot: MvccSnapshot[applied={T|T < 7316332616942166016}]
I20260808 18:33:37.830757 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.827275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.27.251.193:40093)
0808 18:33:37.827416 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.827420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.827432 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:33:37.827550 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.827552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.827598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.827660 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:37.827935 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.828424 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.829692 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.829935 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.830015 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.830309 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:37.830374 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:37.830427 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.830570 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.830613 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:33:37.842566 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.838308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:53105 (local address 127.27.251.193:40093)
0808 18:33:37.838422 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:37.838427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:37.838662 (+ 235us) server_negotiation.cc:407] Connection header received
0808 18:33:37.838745 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:37.838748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:37.838794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:37.838868 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:37.839118 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.839911 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.841321 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.841561 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.841655 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.842034 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:37.842100 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:37.842164 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:37.842286 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 18:33:37.842331 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":43}
I20260808 18:33:37.842984 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:37.838269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.27.251.193:53105)
0808 18:33:37.838584 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:37.838596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:37.838679 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:37.838883 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:37.838885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:37.838891 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:37.839022 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.839027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.840089 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.840092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.841101 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:37.841179 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:37.841701 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:37.841705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:37.841814 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:37.841863 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:37.842190 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:37.842197 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:33:37.842268 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 18:33:37.842356 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":18,"thread_start_us":126,"threads_started":1}
I20260808 18:33:37.849436 31052 raft_consensus.cc:1261] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:37.849475 30893 raft_consensus.cc:1261] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:37.849805 31137 consensus_queue.cc:1035] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:37.850023 31140 consensus_queue.cc:1035] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:37.892251 30564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:37.892427 30564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:37.894870 30564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Sending DeleteTablet for 3 replicas of tablet f4ed1ce9860c45abb465a4e341b1b060
I20260808 18:33:37.895568 30871 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:37 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57410
I20260808 18:33:37.895799 31030 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:37 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:58116
I20260808 18:33:37.896266 31177 tablet_replica.cc:330] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: stopping tablet replica
I20260808 18:33:37.896231 30714 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ed1ce9860c45abb465a4e341b1b060 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:37 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37428
I20260808 18:33:37.896497 30564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:33:37.896755 31176 tablet_replica.cc:330] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: stopping tablet replica
W20260808 18:33:37.896963 30564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:37.896973 31177 raft_consensus.cc:2229] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:37.897058 31176 raft_consensus.cc:2229] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:37.897302 31176 raft_consensus.cc:2258] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:37.897522 31177 raft_consensus.cc:2258] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:37.898532 31178 tablet_replica.cc:330] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: stopping tablet replica
I20260808 18:33:37.898808 31178 raft_consensus.cc:2229] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:37.898860 31176 ts_tablet_manager.cc:1905] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:37.900139 31178 raft_consensus.cc:2258] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:37.900347 31177 ts_tablet_manager.cc:1905] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:37.902220 31176 ts_tablet_manager.cc:1918] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:37.902333 31176 log.cc:1196] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/f4ed1ce9860c45abb465a4e341b1b060
I20260808 18:33:37.902335 31178 ts_tablet_manager.cc:1905] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:37.902449 30714 tablet_service.cc:1467] Processing CreateTablet for tablet e4422cf6c11040c28e579a832d7a328a (DEFAULT_TABLE table=test-table [id=4ea18df33f244eadbf57048e83368b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.902696 31176 ts_tablet_manager.cc:1939] T f4ed1ce9860c45abb465a4e341b1b060 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting consensus metadata
I20260808 18:33:37.902755 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4422cf6c11040c28e579a832d7a328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.903919 31177 ts_tablet_manager.cc:1918] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:37.904074 31177 log.cc:1196] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/f4ed1ce9860c45abb465a4e341b1b060
I20260808 18:33:37.904445 31177 ts_tablet_manager.cc:1939] T f4ed1ce9860c45abb465a4e341b1b060 P 02404116a27d4c86a579a2785e8d4b3b: Deleting consensus metadata
I20260808 18:33:37.905273 31131 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Bootstrap starting.
I20260808 18:33:37.905452 31178 ts_tablet_manager.cc:1918] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:37.905441 31030 tablet_service.cc:1467] Processing CreateTablet for tablet e4422cf6c11040c28e579a832d7a328a (DEFAULT_TABLE table=test-table [id=4ea18df33f244eadbf57048e83368b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.905599 31178 log.cc:1196] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/f4ed1ce9860c45abb465a4e341b1b060
I20260808 18:33:37.905678 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4422cf6c11040c28e579a832d7a328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.905931 31178 ts_tablet_manager.cc:1939] T f4ed1ce9860c45abb465a4e341b1b060 P e900636b10774432ae19cdda9c896a61: Deleting consensus metadata
I20260808 18:33:37.906262 31131 tablet_bootstrap.cc:654] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.907722 31131 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: No bootstrap required, opened a new log
I20260808 18:33:37.907810 31131 ts_tablet_manager.cc:1397] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:33:37.907881 30871 tablet_service.cc:1467] Processing CreateTablet for tablet e4422cf6c11040c28e579a832d7a328a (DEFAULT_TABLE table=test-table [id=4ea18df33f244eadbf57048e83368b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:37.908160 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4422cf6c11040c28e579a832d7a328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:37.908227 31131 raft_consensus.cc:348] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.908349 31131 raft_consensus.cc:374] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.908388 31131 raft_consensus.cc:729] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e900636b10774432ae19cdda9c896a61, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.908499 31131 consensus_queue.cc:260] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.908824 31131 ts_tablet_manager.cc:1428] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:33:37.910425 31129 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Bootstrap starting.
I20260808 18:33:37.911388 31129 tablet_bootstrap.cc:654] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.912081 31128 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Bootstrap starting.
I20260808 18:33:37.912938 31128 tablet_bootstrap.cc:654] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:37.913738 30546 catalog_manager.cc:4841] TS 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843): tablet f4ed1ce9860c45abb465a4e341b1b060 (table test-table [id=cf455183d3984e45bdad1b72f3d71c63]) successfully deleted
I20260808 18:33:37.913839 30549 catalog_manager.cc:4841] TS dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283): tablet f4ed1ce9860c45abb465a4e341b1b060 (table test-table [id=cf455183d3984e45bdad1b72f3d71c63]) successfully deleted
I20260808 18:33:37.913980 30546 catalog_manager.cc:4841] TS e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093): tablet f4ed1ce9860c45abb465a4e341b1b060 (table test-table [id=cf455183d3984e45bdad1b72f3d71c63]) successfully deleted
I20260808 18:33:37.914268 31128 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: No bootstrap required, opened a new log
I20260808 18:33:37.914364 31128 ts_tablet_manager.cc:1397] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 18:33:37.914809 31128 raft_consensus.cc:348] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.914940 31128 raft_consensus.cc:374] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.914990 31128 raft_consensus.cc:729] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd1bd16a9d64891833e04b8839f3c4c, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.915127 31128 consensus_queue.cc:260] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.915402 31128 ts_tablet_manager.cc:1428] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:33:37.916177 31129 tablet_bootstrap.cc:492] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: No bootstrap required, opened a new log
I20260808 18:33:37.916289 31129 ts_tablet_manager.cc:1397] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260808 18:33:37.916930 31129 raft_consensus.cc:348] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.917073 31129 raft_consensus.cc:374] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:37.917114 31129 raft_consensus.cc:729] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Initialized, Role: FOLLOWER
I20260808 18:33:37.917250 31129 consensus_queue.cc:260] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.917532 31129 ts_tablet_manager.cc:1428] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 18:33:37.975172 31173 raft_consensus.cc:482] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:37.975322 31173 raft_consensus.cc:504] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.976039 31173 leader_election.cc:288] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.976624 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4422cf6c11040c28e579a832d7a328a" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" is_pre_election: true
I20260808 18:33:37.976646 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4422cf6c11040c28e579a832d7a328a" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b" is_pre_election: true
I20260808 18:33:37.976807 31052 raft_consensus.cc:2454] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.976809 30893 raft_consensus.cc:2454] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:37.977149 30659 leader_election.cc:302] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.977370 31173 raft_consensus.cc:2781] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:37.977449 31173 raft_consensus.cc:482] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:37.977514 31173 raft_consensus.cc:3037] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.978586 31173 raft_consensus.cc:504] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.978999 31173 leader_election.cc:288] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Requested vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:37.979521 31052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4422cf6c11040c28e579a832d7a328a" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b"
I20260808 18:33:37.979571 30893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4422cf6c11040c28e579a832d7a328a" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
I20260808 18:33:37.979636 31052 raft_consensus.cc:3037] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.979693 30893 raft_consensus.cc:3037] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:37.980808 31052 raft_consensus.cc:2454] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.980957 30893 raft_consensus.cc:2454] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:37.981141 30659 leader_election.cc:302] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:37.981324 31173 raft_consensus.cc:2781] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:37.981482 31173 raft_consensus.cc:686] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Becoming Leader. State: Replica: e900636b10774432ae19cdda9c896a61, State: Running, Role: LEADER
I20260808 18:33:37.981663 31173 consensus_queue.cc:237] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:37.983560 30561 catalog_manager.cc:5495] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e900636b10774432ae19cdda9c896a61 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:38.015295 31050 raft_consensus.cc:1261] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:38.015686 30891 raft_consensus.cc:1261] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:38.015730 31169 consensus_queue.cc:1035] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:38.016079 31169 consensus_queue.cc:1035] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:33:38.034382 31157 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 18:33:39.589468 30561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:39.589644 30561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:39.591849 30561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Sending DeleteTablet for 3 replicas of tablet e4422cf6c11040c28e579a832d7a328a
I20260808 18:33:39.592368 30714 tablet_service.cc:1514] Processing DeleteTablet for tablet e4422cf6c11040c28e579a832d7a328a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37428
I20260808 18:33:39.592509 30871 tablet_service.cc:1514] Processing DeleteTablet for tablet e4422cf6c11040c28e579a832d7a328a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57410
I20260808 18:33:39.592535 31030 tablet_service.cc:1514] Processing DeleteTablet for tablet e4422cf6c11040c28e579a832d7a328a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:58116
I20260808 18:33:39.592891 31209 tablet_replica.cc:330] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: stopping tablet replica
I20260808 18:33:39.592892 31207 tablet_replica.cc:330] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: stopping tablet replica
I20260808 18:33:39.592904 31208 tablet_replica.cc:330] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: stopping tablet replica
I20260808 18:33:39.593024 31208 raft_consensus.cc:2229] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:39.593024 31207 raft_consensus.cc:2229] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:39.593145 31208 raft_consensus.cc:2258] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.593225 31207 raft_consensus.cc:2258] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.593148 31209 raft_consensus.cc:2229] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:39.594024 31209 raft_consensus.cc:2258] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.594805 31209 ts_tablet_manager.cc:1905] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.595739 31207 ts_tablet_manager.cc:1905] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.595894 31208 ts_tablet_manager.cc:1905] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.597834 31209 ts_tablet_manager.cc:1918] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:33:39.597936 31209 log.cc:1196] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/e4422cf6c11040c28e579a832d7a328a
I20260808 18:33:39.598192 31209 ts_tablet_manager.cc:1939] T e4422cf6c11040c28e579a832d7a328a P 02404116a27d4c86a579a2785e8d4b3b: Deleting consensus metadata
I20260808 18:33:39.598574 31207 ts_tablet_manager.cc:1918] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:33:39.598564 31208 ts_tablet_manager.cc:1918] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:33:39.598652 31208 log.cc:1196] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/e4422cf6c11040c28e579a832d7a328a
I20260808 18:33:39.598652 31207 log.cc:1196] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/e4422cf6c11040c28e579a832d7a328a
I20260808 18:33:39.598933 31208 ts_tablet_manager.cc:1939] T e4422cf6c11040c28e579a832d7a328a P dcd1bd16a9d64891833e04b8839f3c4c: Deleting consensus metadata
I20260808 18:33:39.598948 31207 ts_tablet_manager.cc:1939] T e4422cf6c11040c28e579a832d7a328a P e900636b10774432ae19cdda9c896a61: Deleting consensus metadata
I20260808 18:33:39.598901 30546 catalog_manager.cc:4841] TS 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843): tablet e4422cf6c11040c28e579a832d7a328a (table test-table [id=4ea18df33f244eadbf57048e83368b19]) successfully deleted
I20260808 18:33:39.599664 30549 catalog_manager.cc:4841] TS dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283): tablet e4422cf6c11040c28e579a832d7a328a (table test-table [id=4ea18df33f244eadbf57048e83368b19]) successfully deleted
I20260808 18:33:39.599877 30546 catalog_manager.cc:4841] TS e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093): tablet e4422cf6c11040c28e579a832d7a328a (table test-table [id=4ea18df33f244eadbf57048e83368b19]) successfully deleted
Aug 08 18:33:39 dist-test-slave-fd7l krb5kdc[30513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:33:39 dist-test-slave-fd7l krb5kdc[30513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:39.631603 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.617253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.27.251.254:38795)
0808 18:33:39.617596 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.617608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.617675 (+ 67us) server_negotiation.cc:407] Connection header received
0808 18:33:39.617876 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.617879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.617926 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.618039 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:39.618783 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.619357 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.620215 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.620453 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.620628 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.628176 (+ 7548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:39.628180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:39.628182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:39.628211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:39.629846 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:39.630286 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:39.630290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:39.630292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:39.630390 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:39.630680 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:39.630683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:39.630685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:39.630885 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:39.631031 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.631282 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.631415 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260808 18:33:39.634783 30561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:39.635356 30561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:39.640702 30871 tablet_service.cc:1467] Processing CreateTablet for tablet 65be2fd471574681b25d675aac997025 (DEFAULT_TABLE table=test-table [id=59268ae2f4f841098a16f9d8f04d40e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.640673 30714 tablet_service.cc:1467] Processing CreateTablet for tablet 65be2fd471574681b25d675aac997025 (DEFAULT_TABLE table=test-table [id=59268ae2f4f841098a16f9d8f04d40e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.640763 31030 tablet_service.cc:1467] Processing CreateTablet for tablet 65be2fd471574681b25d675aac997025 (DEFAULT_TABLE table=test-table [id=59268ae2f4f841098a16f9d8f04d40e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.641026 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65be2fd471574681b25d675aac997025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.641026 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65be2fd471574681b25d675aac997025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.641047 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65be2fd471574681b25d675aac997025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.643941 31219 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Bootstrap starting.
I20260808 18:33:39.644294 31220 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Bootstrap starting.
I20260808 18:33:39.644724 31218 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Bootstrap starting.
I20260808 18:33:39.644966 31219 tablet_bootstrap.cc:654] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.645398 31220 tablet_bootstrap.cc:654] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.646081 31218 tablet_bootstrap.cc:654] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.646436 31219 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: No bootstrap required, opened a new log
I20260808 18:33:39.646556 31219 ts_tablet_manager.cc:1397] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:39.646996 31220 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: No bootstrap required, opened a new log
I20260808 18:33:39.647053 31219 raft_consensus.cc:348] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.647167 31220 ts_tablet_manager.cc:1397] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:39.647188 31219 raft_consensus.cc:374] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.647233 31219 raft_consensus.cc:729] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e900636b10774432ae19cdda9c896a61, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.647495 31218 tablet_bootstrap.cc:492] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: No bootstrap required, opened a new log
I20260808 18:33:39.647457 31219 consensus_queue.cc:260] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.647624 31218 ts_tablet_manager.cc:1397] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:39.647676 31220 raft_consensus.cc:348] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.647816 31220 raft_consensus.cc:374] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.647862 31220 raft_consensus.cc:729] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.647992 31220 consensus_queue.cc:260] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.648160 31218 raft_consensus.cc:348] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.648279 31218 raft_consensus.cc:374] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.648326 31218 raft_consensus.cc:729] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd1bd16a9d64891833e04b8839f3c4c, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.648476 31220 ts_tablet_manager.cc:1428] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:33:39.648463 31218 consensus_queue.cc:260] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.648892 31218 ts_tablet_manager.cc:1428] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:33:39.649693 31219 ts_tablet_manager.cc:1428] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:39.673426 31225 raft_consensus.cc:482] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:39.673616 31225 raft_consensus.cc:504] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.674823 31225 leader_election.cc:288] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093), dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283)
I20260808 18:33:39.679247 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.675271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:47773 (local address 127.27.251.194:44283)
0808 18:33:39.675552 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.675562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.675736 (+ 174us) server_negotiation.cc:407] Connection header received
0808 18:33:39.675910 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.675913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.675957 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.676053 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:39.676368 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.676970 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678163 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.678589 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678723 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.678782 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.679011 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.679091 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
I20260808 18:33:39.679373 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.675176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:48679 (local address 127.27.251.193:40093)
0808 18:33:39.675486 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.675501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.675619 (+ 118us) server_negotiation.cc:407] Connection header received
0808 18:33:39.675809 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.675812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.675859 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.675957 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:39.676275 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.676959 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678179 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.678659 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678812 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.678873 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.679108 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.679191 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
I20260808 18:33:39.679244 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.675116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:44283 (local address 127.27.251.195:47773)
0808 18:33:39.675631 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:39.675642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:39.675779 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:39.676065 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:39.676068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:39.676113 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:39.676245 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.676250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.677118 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.677122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.678006 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.678015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.678727 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.678882 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.678902 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 18:33:39.678992 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 18:33:39.679062 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":99,"thread_start_us":134,"threads_started":1}
I20260808 18:33:39.679419 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.675015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.27.251.195:48679)
0808 18:33:39.675484 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:39.675502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:39.675680 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:39.675980 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:39.675984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:39.676002 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:39.676151 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.676158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.677090 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.677096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.678038 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.678048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.678821 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.678825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.678967 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.678986 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:33:39.679087 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 18:33:39.679166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":1,"thread_start_us":153,"threads_started":1}
I20260808 18:33:39.680128 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65be2fd471574681b25d675aac997025" candidate_uuid: "02404116a27d4c86a579a2785e8d4b3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" is_pre_election: true
I20260808 18:33:39.680312 30891 raft_consensus.cc:2454] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02404116a27d4c86a579a2785e8d4b3b in term 0.
I20260808 18:33:39.680437 30734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65be2fd471574681b25d675aac997025" candidate_uuid: "02404116a27d4c86a579a2785e8d4b3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e900636b10774432ae19cdda9c896a61" is_pre_election: true
I20260808 18:33:39.680739 30734 raft_consensus.cc:2454] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02404116a27d4c86a579a2785e8d4b3b in term 0.
I20260808 18:33:39.680766 30976 leader_election.cc:302] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, dcd1bd16a9d64891833e04b8839f3c4c; no voters:
I20260808 18:33:39.681067 31225 raft_consensus.cc:2781] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:39.681136 31225 raft_consensus.cc:482] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:39.681203 31225 raft_consensus.cc:3037] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:39.682442 31225 raft_consensus.cc:504] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.682873 31225 leader_election.cc:288] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [CANDIDATE]: Term 1 election: Requested vote from peers e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093), dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283)
I20260808 18:33:39.683247 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65be2fd471574681b25d675aac997025" candidate_uuid: "02404116a27d4c86a579a2785e8d4b3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
I20260808 18:33:39.683406 30891 raft_consensus.cc:3037] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:39.684481 30891 raft_consensus.cc:2454] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02404116a27d4c86a579a2785e8d4b3b in term 1.
I20260808 18:33:39.684818 30976 leader_election.cc:302] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, dcd1bd16a9d64891833e04b8839f3c4c; no voters:
I20260808 18:33:39.684801 30734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65be2fd471574681b25d675aac997025" candidate_uuid: "02404116a27d4c86a579a2785e8d4b3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e900636b10774432ae19cdda9c896a61"
I20260808 18:33:39.684952 30734 raft_consensus.cc:3037] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:39.685017 31225 raft_consensus.cc:2781] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:39.685285 31225 raft_consensus.cc:686] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 1 LEADER]: Becoming Leader. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Running, Role: LEADER
I20260808 18:33:39.685565 31225 consensus_queue.cc:237] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.686292 30734 raft_consensus.cc:2454] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02404116a27d4c86a579a2785e8d4b3b in term 1.
I20260808 18:33:39.688189 30564 catalog_manager.cc:5495] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195). New cstate: current_term: 1 leader_uuid: "02404116a27d4c86a579a2785e8d4b3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } health_report { overall_health: HEALTHY } } }
I20260808 18:33:39.705649 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.701083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.27.251.195:46843)
0808 18:33:39.701411 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.701421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.701434 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:33:39.701560 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.701563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.701621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.701784 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:39.702090 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.702739 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.703748 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.704059 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.704183 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.704591 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:39.705124 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:39.705272 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.705359 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.705433 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":122,"threads_started":1}
I20260808 18:33:39.716970 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.713335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:59909 (local address 127.27.251.193:40093)
0808 18:33:39.713493 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.713501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.713684 (+ 183us) server_negotiation.cc:407] Connection header received
0808 18:33:39.713797 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.713799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.713848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.713918 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:39.714167 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.714784 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.715767 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.716096 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.716193 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.716458 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:39.716613 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:39.716694 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.716797 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.716848 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:33:39.716981 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.713232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.27.251.195:59909)
0808 18:33:39.713594 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:39.713608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:39.713705 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:39.713937 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:39.713939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:39.713945 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:39.714070 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.714075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.714936 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.714939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.715659 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.715668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.716212 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.716214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.716288 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.716336 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:39.716713 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:39.716720 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:33:39.716780 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 18:33:39.716854 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":124,"threads_started":1}
I20260808 18:33:39.724298 30891 raft_consensus.cc:1261] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Refusing update from remote peer 02404116a27d4c86a579a2785e8d4b3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:39.724442 30734 raft_consensus.cc:1261] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Refusing update from remote peer 02404116a27d4c86a579a2785e8d4b3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:39.724828 31234 consensus_queue.cc:1035] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:39.725011 31225 consensus_queue.cc:1035] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:39.743606 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.739814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46843 (local address 127.27.251.193:38811)
0808 18:33:39.740198 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:39.740214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:39.740313 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:39.740581 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:39.740584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:39.740593 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:33:39.740695 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.740707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.741479 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.741481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.742267 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:39.742276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.742860 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.742868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:39.742983 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.743013 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:33:39.743301 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:33:39.743311 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 18:33:39.743386 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:33:39.743476 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":130,"threads_started":1}
I20260808 18:33:39.743594 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:39.740118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:38811 (local address 127.27.251.195:46843)
0808 18:33:39.740245 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:39.740250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:39.740265 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:33:39.740419 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:39.740421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:39.740464 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:39.740545 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:39.740820 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.741359 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.742386 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:39.742703 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:39.742821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:39.743124 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:39.743209 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:39.743295 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:39.743396 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 18:33:39.743448 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:33:39.765506 30564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:39.765691 30564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:39.767966 30564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Sending DeleteTablet for 3 replicas of tablet 65be2fd471574681b25d675aac997025
I20260808 18:33:39.768898 31030 tablet_service.cc:1514] Processing DeleteTablet for tablet 65be2fd471574681b25d675aac997025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:58116
I20260808 18:33:39.768918 30871 tablet_service.cc:1514] Processing DeleteTablet for tablet 65be2fd471574681b25d675aac997025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57410
I20260808 18:33:39.769114 31209 tablet_replica.cc:330] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: stopping tablet replica
I20260808 18:33:39.769151 31208 tablet_replica.cc:330] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: stopping tablet replica
I20260808 18:33:39.769245 31209 raft_consensus.cc:2229] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:39.769281 31208 raft_consensus.cc:2229] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:39.769465 31208 raft_consensus.cc:2258] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.769570 31209 raft_consensus.cc:2258] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.769543 30714 tablet_service.cc:1514] Processing DeleteTablet for tablet 65be2fd471574681b25d675aac997025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:39 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37428
I20260808 18:33:39.769742 31207 tablet_replica.cc:330] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: stopping tablet replica
I20260808 18:33:39.769852 31207 raft_consensus.cc:2229] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:39.769959 31207 raft_consensus.cc:2258] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:39.770445 31209 ts_tablet_manager.cc:1905] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.770635 31208 ts_tablet_manager.cc:1905] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.770951 31207 ts_tablet_manager.cc:1905] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:39.773288 31209 ts_tablet_manager.cc:1918] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:39.773360 31209 log.cc:1196] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/65be2fd471574681b25d675aac997025
I20260808 18:33:39.773357 31208 ts_tablet_manager.cc:1918] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:39.773430 31208 log.cc:1196] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/65be2fd471574681b25d675aac997025
I20260808 18:33:39.773636 31209 ts_tablet_manager.cc:1939] T 65be2fd471574681b25d675aac997025 P 02404116a27d4c86a579a2785e8d4b3b: Deleting consensus metadata
I20260808 18:33:39.773705 31208 ts_tablet_manager.cc:1939] T 65be2fd471574681b25d675aac997025 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting consensus metadata
I20260808 18:33:39.773875 31207 ts_tablet_manager.cc:1918] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:33:39.773942 31207 log.cc:1196] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/65be2fd471574681b25d675aac997025
I20260808 18:33:39.774206 31207 ts_tablet_manager.cc:1939] T 65be2fd471574681b25d675aac997025 P e900636b10774432ae19cdda9c896a61: Deleting consensus metadata
I20260808 18:33:39.774420 30546 catalog_manager.cc:4841] TS 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843): tablet 65be2fd471574681b25d675aac997025 (table test-table [id=59268ae2f4f841098a16f9d8f04d40e7]) successfully deleted
I20260808 18:33:39.774554 30549 catalog_manager.cc:4841] TS dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283): tablet 65be2fd471574681b25d675aac997025 (table test-table [id=59268ae2f4f841098a16f9d8f04d40e7]) successfully deleted
I20260808 18:33:39.774938 30546 catalog_manager.cc:4841] TS e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093): tablet 65be2fd471574681b25d675aac997025 (table test-table [id=59268ae2f4f841098a16f9d8f04d40e7]) successfully deleted
I20260808 18:33:39.775192 30564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:39.775672 30564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:39.780304 30714 tablet_service.cc:1467] Processing CreateTablet for tablet 86bd518db8574775a53a86bae0d4d603 (DEFAULT_TABLE table=test-table [id=16824bd9233f45698fcdb0f6206b5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.780390 30871 tablet_service.cc:1467] Processing CreateTablet for tablet 86bd518db8574775a53a86bae0d4d603 (DEFAULT_TABLE table=test-table [id=16824bd9233f45698fcdb0f6206b5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.780697 30871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bd518db8574775a53a86bae0d4d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.780665 31030 tablet_service.cc:1467] Processing CreateTablet for tablet 86bd518db8574775a53a86bae0d4d603 (DEFAULT_TABLE table=test-table [id=16824bd9233f45698fcdb0f6206b5805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:39.780896 31030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bd518db8574775a53a86bae0d4d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.781041 30714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86bd518db8574775a53a86bae0d4d603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:39.783258 31220 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Bootstrap starting.
I20260808 18:33:39.783295 31219 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: Bootstrap starting.
I20260808 18:33:39.784053 31219 tablet_bootstrap.cc:654] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.784112 31220 tablet_bootstrap.cc:654] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.784816 31218 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Bootstrap starting.
I20260808 18:33:39.785398 31219 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: No bootstrap required, opened a new log
I20260808 18:33:39.785494 31219 ts_tablet_manager.cc:1397] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 18:33:39.785812 31218 tablet_bootstrap.cc:654] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:39.785977 31219 raft_consensus.cc:348] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.786123 31219 raft_consensus.cc:374] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.786170 31219 raft_consensus.cc:729] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e900636b10774432ae19cdda9c896a61, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.786315 31219 consensus_queue.cc:260] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.786768 31219 ts_tablet_manager.cc:1428] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 18:33:39.787555 31220 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: No bootstrap required, opened a new log
I20260808 18:33:39.787709 31220 ts_tablet_manager.cc:1397] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 18:33:39.788169 31220 raft_consensus.cc:348] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.788307 31220 raft_consensus.cc:374] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.788369 31220 raft_consensus.cc:729] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02404116a27d4c86a579a2785e8d4b3b, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.788542 31220 consensus_queue.cc:260] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.788830 31220 ts_tablet_manager.cc:1428] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:33:39.789768 31218 tablet_bootstrap.cc:492] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: No bootstrap required, opened a new log
I20260808 18:33:39.789866 31218 ts_tablet_manager.cc:1397] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:33:39.790344 31218 raft_consensus.cc:348] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.790473 31218 raft_consensus.cc:374] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:39.790509 31218 raft_consensus.cc:729] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd1bd16a9d64891833e04b8839f3c4c, State: Initialized, Role: FOLLOWER
I20260808 18:33:39.790647 31218 consensus_queue.cc:260] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:39.790957 31218 ts_tablet_manager.cc:1428] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:33:40.012217 31194 raft_consensus.cc:482] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:40.012375 31194 raft_consensus.cc:504] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:40.012928 31194 leader_election.cc:288] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:40.013445 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bd518db8574775a53a86bae0d4d603" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" is_pre_election: true
I20260808 18:33:40.013485 31050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bd518db8574775a53a86bae0d4d603" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b" is_pre_election: true
I20260808 18:33:40.013605 30891 raft_consensus.cc:2454] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:40.013686 31050 raft_consensus.cc:2454] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 0.
I20260808 18:33:40.013949 30662 leader_election.cc:302] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcd1bd16a9d64891833e04b8839f3c4c, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:40.014211 31194 raft_consensus.cc:2781] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:40.014261 31194 raft_consensus.cc:482] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:40.014288 31194 raft_consensus.cc:3037] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:40.015434 31194 raft_consensus.cc:504] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:40.015800 31194 leader_election.cc:288] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Requested vote from peers dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283), 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843)
I20260808 18:33:40.016222 31050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bd518db8574775a53a86bae0d4d603" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02404116a27d4c86a579a2785e8d4b3b"
I20260808 18:33:40.016227 30891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86bd518db8574775a53a86bae0d4d603" candidate_uuid: "e900636b10774432ae19cdda9c896a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd1bd16a9d64891833e04b8839f3c4c"
I20260808 18:33:40.016337 31050 raft_consensus.cc:3037] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:40.016363 30891 raft_consensus.cc:3037] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:40.017470 30891 raft_consensus.cc:2454] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:40.017475 31050 raft_consensus.cc:2454] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e900636b10774432ae19cdda9c896a61 in term 1.
I20260808 18:33:40.017793 30659 leader_election.cc:302] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02404116a27d4c86a579a2785e8d4b3b, e900636b10774432ae19cdda9c896a61; no voters:
I20260808 18:33:40.017983 31194 raft_consensus.cc:2781] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:40.018088 31194 raft_consensus.cc:686] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [term 1 LEADER]: Becoming Leader. State: Replica: e900636b10774432ae19cdda9c896a61, State: Running, Role: LEADER
I20260808 18:33:40.018246 31194 consensus_queue.cc:237] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } }
I20260808 18:33:40.020092 30563 catalog_manager.cc:5495] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to e900636b10774432ae19cdda9c896a61 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "e900636b10774432ae19cdda9c896a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e900636b10774432ae19cdda9c896a61" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:40.039323 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:40.035713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.27.251.193:40093)
0808 18:33:40.035898 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:40.035903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:40.035918 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:33:40.035994 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:40.035997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:40.036077 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:40.036164 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:40.036466 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:40.037217 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:40.038343 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:40.038562 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:40.038644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:40.038842 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:40.038907 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:40.038959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:40.039073 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 18:33:40.039114 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 18:33:40.045469 31050 raft_consensus.cc:1261] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:40.045643 30893 raft_consensus.cc:1261] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Refusing update from remote peer e900636b10774432ae19cdda9c896a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:40.045902 31194 consensus_queue.cc:1035] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02404116a27d4c86a579a2785e8d4b3b" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:40.046092 31169 consensus_queue.cc:1035] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd1bd16a9d64891833e04b8839f3c4c" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:33:40.059902 31186 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 18:33:41.615267 30564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:41.615443 30564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:41.618376 30564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6fdac017694f0e888bb0f02d78213f: Sending DeleteTablet for 3 replicas of tablet 86bd518db8574775a53a86bae0d4d603
I20260808 18:33:41.618943 30714 tablet_service.cc:1514] Processing DeleteTablet for tablet 86bd518db8574775a53a86bae0d4d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37428
I20260808 18:33:41.619055 30871 tablet_service.cc:1514] Processing DeleteTablet for tablet 86bd518db8574775a53a86bae0d4d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57410
I20260808 18:33:41.619374 31278 tablet_replica.cc:330] T 86bd518db8574775a53a86bae0d4d603 P e900636b10774432ae19cdda9c896a61: stopping tablet replica
I20260808 18:33:41.619364 31030 tablet_service.cc:1514] Processing DeleteTablet for tablet 86bd518db8574775a53a86bae0d4d603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:58116
I20260808 18:33:41.619449 31279 tablet_replica.cc:330] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: stopping tablet replica
I20260808 18:33:41.619522 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30643
I20260808 18:33:41.622797 31279 raft_consensus.cc:2229] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:41.622988 31279 raft_consensus.cc:2258] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:41.623813 31279 ts_tablet_manager.cc:1905] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:41.624716 31280 tablet_replica.cc:330] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: stopping tablet replica
I20260808 18:33:41.624899 31280 raft_consensus.cc:2229] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:41.625059 31280 raft_consensus.cc:2258] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:41.625706 31280 ts_tablet_manager.cc:1905] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:41.627698 31279 ts_tablet_manager.cc:1918] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:33:41.627796 31279 log.cc:1196] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/86bd518db8574775a53a86bae0d4d603
I20260808 18:33:41.627892 31280 ts_tablet_manager.cc:1918] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 18:33:41.628005 31280 log.cc:1196] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/86bd518db8574775a53a86bae0d4d603
I20260808 18:33:41.628095 31279 ts_tablet_manager.cc:1939] T 86bd518db8574775a53a86bae0d4d603 P dcd1bd16a9d64891833e04b8839f3c4c: Deleting consensus metadata
I20260808 18:33:41.628346 31280 ts_tablet_manager.cc:1939] T 86bd518db8574775a53a86bae0d4d603 P 02404116a27d4c86a579a2785e8d4b3b: Deleting consensus metadata
I20260808 18:33:41.629154 30549 catalog_manager.cc:4841] TS dcd1bd16a9d64891833e04b8839f3c4c (127.27.251.194:44283): tablet 86bd518db8574775a53a86bae0d4d603 (table test-table [id=16824bd9233f45698fcdb0f6206b5805]) successfully deleted
I20260808 18:33:41.629197 30546 catalog_manager.cc:4841] TS 02404116a27d4c86a579a2785e8d4b3b (127.27.251.195:46843): tablet 86bd518db8574775a53a86bae0d4d603 (table test-table [id=16824bd9233f45698fcdb0f6206b5805]) successfully deleted
W20260808 18:33:41.633466 30546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:40093 (error 108)
I20260808 18:33:41.633838 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30798
I20260808 18:33:41.635533 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:41.635106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40093 (local address 127.0.0.1:37476)
0808 18:33:41.635367 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:41.635442 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
W20260808 18:33:41.639534 30546 catalog_manager.cc:4584] TS e900636b10774432ae19cdda9c896a61 (127.27.251.193:40093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86bd518db8574775a53a86bae0d4d603: Network error: Client connection negotiation failed: client connection to 127.27.251.193:40093: connect: Connection refused (error 111)
I20260808 18:33:41.642341 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30956
I20260808 18:33:41.652021 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 30529
2026-08-08T18:33:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6507 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:41 dist-test-slave-fd7l krb5kdc[31285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:41 dist-test-slave-fd7l krb5kdc[31285](info): set up 2 sockets
Aug 08 18:33:41 dist-test-slave-fd7l krb5kdc[31285](info): commencing operation
krb5kdc: starting...
W20260808 18:33:42.747591 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:42 dist-test-slave-fd7l krb5kdc[31285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:42.876353 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:46425
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:46425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:42.998164 31300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:42.998450 31300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:42.998503 31300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:43.002537 31300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:43.002600 31300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:43.002620 31300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:43.002638 31300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:43.002653 31300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:43.008013 31300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:46425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:43.009426 31300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:43.010462 31300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:43.020131 31300 server_base.cc:1034] running on GCE node
W20260808 18:33:43.020212 31311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.020157 31308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.020218 31307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.021250 31300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:43.022590 31300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:43.023809 31300 hybrid_clock.cc:648] HybridClock initialized: now 1786214023023794 us; error 43 us; skew 500 ppm
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:43.027022 31300 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:43.027984 31300 webserver.cc:460] Webserver started at http://127.27.251.254:34915/ using document root <none> and password file <none>
I20260808 18:33:43.028721 31300 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:43.028820 31300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:43.029052 31300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:43.031189 31300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "69f982074ca04be3a3ff894f41b45218"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "ebfb0866606c560528fdebd1c61ccf27"
server_key_iv: "08f5823c0bba7a82a39d66b4e40cfb17"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.031788 31300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "69f982074ca04be3a3ff894f41b45218"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "ebfb0866606c560528fdebd1c61ccf27"
server_key_iv: "08f5823c0bba7a82a39d66b4e40cfb17"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.035697 31300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:43.038240 31318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.039453 31300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:43.039646 31300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "69f982074ca04be3a3ff894f41b45218"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "ebfb0866606c560528fdebd1c61ccf27"
server_key_iv: "08f5823c0bba7a82a39d66b4e40cfb17"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.039765 31300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:43.054714 31300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:43.057991 31300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:43.058238 31300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:43.066023 31300 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:46425
I20260808 18:33:43.066111 31377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:46425 every 8 connection(s)
I20260808 18:33:43.067112 31300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:43.070118 31378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:43.074324 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 31300
I20260808 18:33:43.074469 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:43.074739 28655 external_mini_cluster.cc:1428] Setting key c1d1224c4a467c2f02d7c1fbec36e50d
I20260808 18:33:43.076402 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218: Bootstrap starting.
I20260808 18:33:43.079030 31378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:43.079977 31378 log.cc:824] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:43.082254 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218: No bootstrap required, opened a new log
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:43.086593 31378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } }
I20260808 18:33:43.086822 31378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:43.086881 31378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f982074ca04be3a3ff894f41b45218, State: Initialized, Role: FOLLOWER
I20260808 18:33:43.087329 31378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } }
I20260808 18:33:43.087461 31378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:43.087558 31378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:43.087659 31378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:43.088937 31378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } }
I20260808 18:33:43.089313 31378 leader_election.cc:302] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f982074ca04be3a3ff894f41b45218; no voters:
I20260808 18:33:43.089694 31378 leader_election.cc:288] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:43.089811 31383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:43.090046 31383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [term 1 LEADER]: Becoming Leader. State: Replica: 69f982074ca04be3a3ff894f41b45218, State: Running, Role: LEADER
I20260808 18:33:43.090387 31383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } }
I20260808 18:33:43.091045 31378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:43.091502 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.076324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.27.251.254:46425)
0808 18:33:43.076800 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:43.076816 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:43.076865 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:33:43.077563 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:43.077582 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:43.077935 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:43.078229 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:43.079269 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.080799 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.081747 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.082449 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.082635 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.085596 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:43.085602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:43.085618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:43.085647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:43.087779 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.088297 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.088302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.088307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.088432 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.088785 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.088788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.088789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.089304 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:43.089507 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:43.090009 (+ 502us) server_negotiation.cc:299] Negotiation successful
0808 18:33:43.090416 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":182,"threads_started":1}
I20260808 18:33:43.092253 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f982074ca04be3a3ff894f41b45218. Latest consensus state: current_term: 1 leader_uuid: "69f982074ca04be3a3ff894f41b45218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } } }
I20260808 18:33:43.092386 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:43.092849 31393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:43.092931 31386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f982074ca04be3a3ff894f41b45218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f982074ca04be3a3ff894f41b45218" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 46425 } } }
I20260808 18:33:43.093050 31386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:43.096048 31393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:43.101770 31393 catalog_manager.cc:1347] Generated new cluster ID: ed358f270ae349a3b460a3ca390df2d3
I20260808 18:33:43.101841 31393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:43.113209 31393 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:43.114495 31393 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:43.131053 31393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f982074ca04be3a3ff894f41b45218: Generated new TSK 0
I20260808 18:33:43.131992 31393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:43.167773 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:43.293368 31409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:43.293668 31409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:43.293742 31409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:43.297768 31409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:43.297849 31409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:43.297945 31409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:43.303238 31409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:43.304358 31409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:43.305456 31409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:43.315907 31415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.315913 31417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.316036 31409 server_base.cc:1034] running on GCE node
W20260808 18:33:43.316215 31414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.316684 31409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:43.317386 31409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:43.318557 31409 hybrid_clock.cc:648] HybridClock initialized: now 1786214023318539 us; error 26 us; skew 500 ppm
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:43.321396 31409 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:43.322300 31409 webserver.cc:460] Webserver started at http://127.27.251.193:34191/ using document root <none> and password file <none>
I20260808 18:33:43.322872 31409 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:43.322963 31409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:43.323211 31409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:43.325388 31409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "dba23c8074f24578868f186b14e962b5"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "cfb6d522d313c37b946d434bb9ba7690"
server_key_iv: "ff1d611b492b05b52b44c96dc3ccb07b"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.325950 31409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "dba23c8074f24578868f186b14e962b5"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "cfb6d522d313c37b946d434bb9ba7690"
server_key_iv: "ff1d611b492b05b52b44c96dc3ccb07b"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.329708 31409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:43.332113 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.333389 31409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:43.333531 31409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "dba23c8074f24578868f186b14e962b5"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "cfb6d522d313c37b946d434bb9ba7690"
server_key_iv: "ff1d611b492b05b52b44c96dc3ccb07b"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.333642 31409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:43.349799 31409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:43.352993 31409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:43.353240 31409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:43.353884 31409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:43.354745 31409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:43.354817 31409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.354890 31409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:43.354943 31409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.364501 31409 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:41367
I20260808 18:33:43.364549 31550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:41367 every 8 connection(s)
I20260808 18:33:43.365499 31409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:43.375744 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 31409
I20260808 18:33:43.375873 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:43.376237 28655 external_mini_cluster.cc:1428] Setting key e59cff08f939e951be47696193905cba
I20260808 18:33:43.379554 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.367418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:59577 (local address 127.27.251.254:46425)
0808 18:33:43.367570 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:43.367574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:43.368632 (+ 1058us) server_negotiation.cc:407] Connection header received
0808 18:33:43.369532 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:43.369535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:43.369581 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:43.369655 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:43.370824 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.371379 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.372564 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.372795 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.372879 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.375411 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:43.375414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:43.375416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:43.375465 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:43.377111 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.377692 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.377696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.377698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.377793 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.378222 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.378225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.378226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.378405 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:43.378483 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:43.379188 (+ 705us) server_negotiation.cc:299] Negotiation successful
0808 18:33:43.379319 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:33:43.380740 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.367772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:46425 (local address 127.27.251.193:59577)
0808 18:33:43.368440 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:43.368483 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:43.369349 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:43.369819 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:43.369825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:43.370198 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:43.370683 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.370696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.371505 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.371508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.372426 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.372436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.372931 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.372934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.373109 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.373685 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:43.373702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:43.375246 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:43.377231 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.377236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.377255 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.377562 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.377915 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.377919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.377921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.378120 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.378526 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:43.378536 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:43.378905 (+ 369us) client_negotiation.cc:769] Sending connection context
0808 18:33:43.379135 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 18:33:43.379516 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":118,"threads_started":1}
I20260808 18:33:43.382079 31551 heartbeater.cc:347] Connected to a master server at 127.27.251.254:46425
I20260808 18:33:43.382351 31551 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:43.383021 31551 heartbeater.cc:511] Master 127.27.251.254:46425 requested a full tablet report, sending...
I20260808 18:33:43.384683 31337 ts_manager.cc:194] Registered new tserver with Master: dba23c8074f24578868f186b14e962b5 (127.27.251.193:41367)
I20260808 18:33:43.385864 31337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:59577
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:43.407959 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:43.529069 31558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:43.529359 31558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:43.529403 31558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:43.533519 31558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:43.533595 31558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:43.533694 31558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:43.539273 31558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:43.540453 31558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:43.541699 31558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:43.552124 31564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.552167 31567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.552245 31558 server_base.cc:1034] running on GCE node
W20260808 18:33:43.552139 31563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.552860 31558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:43.553637 31558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:43.554805 31558 hybrid_clock.cc:648] HybridClock initialized: now 1786214023554788 us; error 30 us; skew 500 ppm
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:43.557956 31558 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:43.558910 31558 webserver.cc:460] Webserver started at http://127.27.251.194:41985/ using document root <none> and password file <none>
I20260808 18:33:43.559512 31558 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:43.559582 31558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:43.559883 31558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:43.562217 31558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "91b4ff4803924cf5a9461fdb925bfcb6"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "6b43f796c4a8a648e4644e9c5dfe9803"
server_key_iv: "ceef40064a570cd7fdc9e66f9d8fbe11"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.562832 31558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "91b4ff4803924cf5a9461fdb925bfcb6"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "6b43f796c4a8a648e4644e9c5dfe9803"
server_key_iv: "ceef40064a570cd7fdc9e66f9d8fbe11"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.566938 31558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:43.569491 31576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.570881 31558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:33:43.571048 31558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "91b4ff4803924cf5a9461fdb925bfcb6"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "6b43f796c4a8a648e4644e9c5dfe9803"
server_key_iv: "ceef40064a570cd7fdc9e66f9d8fbe11"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.571170 31558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:43.581928 31558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:43.584915 31558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:43.585122 31558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:43.585714 31558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:43.586570 31558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:43.586639 31558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.586710 31558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:43.586762 31558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.596045 31558 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:41651
I20260808 18:33:43.596113 31705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:41651 every 8 connection(s)
I20260808 18:33:43.597052 31558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:43.605499 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 31558
I20260808 18:33:43.605633 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:43.605980 28655 external_mini_cluster.cc:1428] Setting key 4169ddbcee828c62ce4e64b677d4b229
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:43.611914 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.598983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:47089 (local address 127.27.251.254:46425)
0808 18:33:43.599139 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:43.599144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:43.600252 (+ 1108us) server_negotiation.cc:407] Connection header received
0808 18:33:43.601197 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:43.601200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:43.601248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:43.601331 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:43.602507 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.603071 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.604163 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.604376 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.604452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.607231 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:43.607235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:43.607237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:43.607262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:43.609094 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.609784 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.609788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.609790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.609910 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.610538 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.610542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.610544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.610750 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:43.610875 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:43.611631 (+ 756us) server_negotiation.cc:299] Negotiation successful
0808 18:33:43.611747 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:33:43.612975 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.599376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:46425 (local address 127.27.251.194:47089)
0808 18:33:43.600095 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:43.600137 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:43.600970 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:43.601491 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:43.601498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:43.601849 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:43.602357 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.602370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.603189 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.603191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.604050 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.604059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.604505 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.604548 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.604698 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.605295 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:43.605318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:43.607049 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:43.609248 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.609253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.609269 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.609667 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.610067 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.610074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.610075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.610354 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.610910 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:43.610918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:43.611321 (+ 403us) client_negotiation.cc:769] Sending connection context
0808 18:33:43.611584 (+ 263us) client_negotiation.cc:240] Negotiation successful
0808 18:33:43.611998 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":116,"threads_started":1}
I20260808 18:33:43.614210 31706 heartbeater.cc:347] Connected to a master server at 127.27.251.254:46425
I20260808 18:33:43.614491 31706 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:43.615181 31706 heartbeater.cc:511] Master 127.27.251.254:46425 requested a full tablet report, sending...
I20260808 18:33:43.616447 31337 ts_manager.cc:194] Registered new tserver with Master: 91b4ff4803924cf5a9461fdb925bfcb6 (127.27.251.194:41651)
I20260808 18:33:43.617079 31337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:47089
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:43.639146 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:43.756808 31711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:43.757074 31711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:43.757119 31711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:43.761174 31711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:43.761236 31711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:43.761294 31711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:43.766786 31711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:43.767812 31711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:43.768827 31711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:43.779232 31716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.779227 31717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:43.779227 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:43.779461 31711 server_base.cc:1034] running on GCE node
I20260808 18:33:43.779995 31711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:43.780728 31711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:43.781927 31711 hybrid_clock.cc:648] HybridClock initialized: now 1786214023781888 us; error 50 us; skew 500 ppm
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:43.784823 31711 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:43.785715 31711 webserver.cc:460] Webserver started at http://127.27.251.195:34099/ using document root <none> and password file <none>
I20260808 18:33:43.786307 31711 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:43.786399 31711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:43.786656 31711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:43.788879 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "46559751198046c68cf4f9f65aa93c45"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "a4a0e436e6f7883ef670ac39815f6376"
server_key_iv: "0532c2ff20b636838cf54e8a5ca5840a"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.789453 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "46559751198046c68cf4f9f65aa93c45"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "a4a0e436e6f7883ef670ac39815f6376"
server_key_iv: "0532c2ff20b636838cf54e8a5ca5840a"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.793504 31711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:43.795953 31727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.797276 31711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:43.797426 31711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "46559751198046c68cf4f9f65aa93c45"
format_stamp: "Formatted at 2026-08-08 18:33:43 on dist-test-slave-fd7l"
server_key: "a4a0e436e6f7883ef670ac39815f6376"
server_key_iv: "0532c2ff20b636838cf54e8a5ca5840a"
server_key_version: "encryptionkey@0"
I20260808 18:33:43.797540 31711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:43.818768 31711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:43.821878 31711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:43.822099 31711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:43.822733 31711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:43.823590 31711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:43.823660 31711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.823736 31711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:43.823787 31711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:43.833613 31711 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:35827
I20260808 18:33:43.833694 31855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:35827 every 8 connection(s)
I20260808 18:33:43.834563 31711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:43.836886 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 31711
I20260808 18:33:43.837016 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:43.837393 28655 external_mini_cluster.cc:1428] Setting key 8e8ace1cccdda214dc5a8613ab75495c
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214023, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:43.849038 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.836563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:34943 (local address 127.27.251.254:46425)
0808 18:33:43.836742 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:43.836749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:43.838100 (+ 1351us) server_negotiation.cc:407] Connection header received
0808 18:33:43.839213 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:43.839216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:43.839277 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:43.839365 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:43.840688 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.841290 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.842580 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.842820 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.842936 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.845581 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:43.845589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:43.845592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:43.845617 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:43.846865 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.847411 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.847414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.847416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.847511 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:43.847876 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:43.847879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:43.847881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:43.848063 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:43.848174 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:43.848729 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 18:33:43.848862 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:33:43.849910 31858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:43.837118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:46425 (local address 127.27.251.195:34943)
0808 18:33:43.837946 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:43.837991 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:43.838996 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:43.839549 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:43.839556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:43.839986 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:43.840497 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.840556 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.841429 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.841433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.842444 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:43.842455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:43.842981 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:43.842985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:43.843164 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:43.843891 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:43.843907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:43.845404 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:43.846988 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.846994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.847010 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.847291 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.847611 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:43.847615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:43.847617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:43.847769 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:43.848173 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:43.848181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:43.848431 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 18:33:43.848669 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 18:33:43.849022 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":164,"threads_started":1}
I20260808 18:33:43.851205 31856 heartbeater.cc:347] Connected to a master server at 127.27.251.254:46425
I20260808 18:33:43.851465 31856 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:43.852059 31856 heartbeater.cc:511] Master 127.27.251.254:46425 requested a full tablet report, sending...
I20260808 18:33:43.853188 31336 ts_manager.cc:194] Registered new tserver with Master: 46559751198046c68cf4f9f65aa93c45 (127.27.251.195:35827)
I20260808 18:33:43.853842 31336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:34943
I20260808 18:33:43.863682 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:33:43.878957 31381 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:43.877406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.27.251.254:46425)
0808 18:33:43.877633 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:43.877636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:43.877712 (+ 76us) server_negotiation.cc:407] Connection header received
0808 18:33:43.877826 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:43.877828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:43.877875 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:43.877950 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:43.878795 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39248: BlockingRecv error: recv got EOF from 127.0.0.1:39248 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
I20260808 18:33:43.880038 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 31409
I20260808 18:33:43.888034 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 31558
I20260808 18:33:43.894726 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 31711
I20260808 18:33:43.902645 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 31300
2026-08-08T18:33:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2250 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31870](info): set up 2 sockets
Aug 08 18:33:43 dist-test-slave-fd7l krb5kdc[31870](info): commencing operation
krb5kdc: starting...
W20260808 18:33:45.022795 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:45.155614 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33379
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:33379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:45.272264 31884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:45.272608 31884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:45.272723 31884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:45.276818 31884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:45.276921 31884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:45.276959 31884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:45.276997 31884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:45.277032 31884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:45.282397 31884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:33379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:45.283548 31884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:45.284657 31884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:45.293308 31889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:45.293349 31892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:45.293514 31884 server_base.cc:1034] running on GCE node
W20260808 18:33:45.293315 31890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:45.294189 31884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:45.295406 31884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:45.296597 31884 hybrid_clock.cc:648] HybridClock initialized: now 1786214025296563 us; error 45 us; skew 500 ppm
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214025, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:45.303716 31884 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:45.304780 31884 webserver.cc:460] Webserver started at http://127.27.251.254:45897/ using document root <none> and password file <none>
I20260808 18:33:45.305423 31884 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:45.305518 31884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:45.305775 31884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:45.307976 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "82e31decb6e34ba893ba56aeac53e44b"
format_stamp: "Formatted at 2026-08-08 18:33:45 on dist-test-slave-fd7l"
server_key: "353bd4b165e4250a7d85daf0ed2b70c6"
server_key_iv: "03ae3f960677e56bd72a85101272f2c8"
server_key_version: "encryptionkey@0"
I20260808 18:33:45.308656 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "82e31decb6e34ba893ba56aeac53e44b"
format_stamp: "Formatted at 2026-08-08 18:33:45 on dist-test-slave-fd7l"
server_key: "353bd4b165e4250a7d85daf0ed2b70c6"
server_key_iv: "03ae3f960677e56bd72a85101272f2c8"
server_key_version: "encryptionkey@0"
I20260808 18:33:45.312669 31884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 18:33:45.315084 31900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:45.316490 31884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:45.316707 31884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "82e31decb6e34ba893ba56aeac53e44b"
format_stamp: "Formatted at 2026-08-08 18:33:45 on dist-test-slave-fd7l"
server_key: "353bd4b165e4250a7d85daf0ed2b70c6"
server_key_iv: "03ae3f960677e56bd72a85101272f2c8"
server_key_version: "encryptionkey@0"
I20260808 18:33:45.316823 31884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:45.347501 31884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:45.350739 31884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:45.350956 31884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:45.358353 31884 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:33379
I20260808 18:33:45.358647 31958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:33379 every 8 connection(s)
I20260808 18:33:45.359403 31884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:45.362401 31959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:45.363588 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 31884
I20260808 18:33:45.363736 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:45.364059 28655 external_mini_cluster.cc:1428] Setting key 1f11fe9b4fce0f2057aff0dac7015aec
I20260808 18:33:45.369514 31959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: Bootstrap starting.
I20260808 18:33:45.372305 31959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:45.373135 31959 log.cc:824] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:45.375615 31959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: No bootstrap required, opened a new log
I20260808 18:33:45.379377 31959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } }
I20260808 18:33:45.379644 31959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:45.379737 31959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e31decb6e34ba893ba56aeac53e44b, State: Initialized, Role: FOLLOWER
I20260808 18:33:45.379812 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:45.365414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.27.251.254:33379)
0808 18:33:45.365862 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:45.365882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:45.365937 (+ 55us) server_negotiation.cc:407] Connection header received
0808 18:33:45.366825 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:45.366848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:45.367282 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:45.367659 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:45.368384 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:45.369894 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:45.370801 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:45.371478 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:45.371651 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:45.374459 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:45.374466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:45.374482 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:45.374513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:45.376787 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:45.377273 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:45.377279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:45.377284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:45.377400 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:45.377705 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:45.377709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:45.377711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:45.378126 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:45.378356 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:45.378643 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 18:33:45.378961 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":137,"threads_started":1}
I20260808 18:33:45.380290 31959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } }
I20260808 18:33:45.380455 31959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:45.380541 31959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:45.380650 31959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:45.381705 31959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } }
I20260808 18:33:45.382081 31959 leader_election.cc:302] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e31decb6e34ba893ba56aeac53e44b; no voters:
I20260808 18:33:45.382452 31959 leader_election.cc:288] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:45.382534 31965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:45.382752 31965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [term 1 LEADER]: Becoming Leader. State: Replica: 82e31decb6e34ba893ba56aeac53e44b, State: Running, Role: LEADER
I20260808 18:33:45.383065 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:45.383061 31965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } }
I20260808 18:33:45.383836 31959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:45.384262 31965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e31decb6e34ba893ba56aeac53e44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } } }
I20260808 18:33:45.384379 31965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:45.385737 31967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e31decb6e34ba893ba56aeac53e44b. Latest consensus state: current_term: 1 leader_uuid: "82e31decb6e34ba893ba56aeac53e44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e31decb6e34ba893ba56aeac53e44b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33379 } } }
I20260808 18:33:45.385855 31967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b [sys.catalog]: This master's current role is: LEADER
W20260808 18:33:45.389422 31983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:45.389515 31983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:45.389638 31970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:45.390543 31970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:45.397518 31970 catalog_manager.cc:1347] Generated new cluster ID: 61a71119453742d0afb312e0655efca8
I20260808 18:33:45.397608 31970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:45.403714 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:45.398314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.27.251.254:33379)
0808 18:33:45.398451 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:45.398456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:45.398881 (+ 425us) server_negotiation.cc:407] Connection header received
0808 18:33:45.399053 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:45.399056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:45.399112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:45.399196 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:45.399770 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:45.400306 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:45.401245 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:45.401539 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:45.401762 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:45.402716 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:45.402720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:45.402723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:45.403211 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:45.403243 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:45.403259 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:45.403371 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:45.403492 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 18:33:45.403561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:33:45.427750 31970 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:45.428966 31970 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:45.439497 31970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e31decb6e34ba893ba56aeac53e44b: Generated new TSK 0
I20260808 18:33:45.440207 31970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:45.451406 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 31884
2026-08-08T18:33:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1546 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31996](info): set up 2 sockets
Aug 08 18:33:45 dist-test-slave-fd7l krb5kdc[31996](info): commencing operation
krb5kdc: starting...
W20260808 18:33:46.579082 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:46 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:46.711366 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41917
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:46.827956 32010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:46.828205 32010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:46.828248 32010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:46.832204 32010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:46.832257 32010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:46.832275 32010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:46.832293 32010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:46.832307 32010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:46.837452 32010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:46.838564 32010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:46.839579 32010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:46.849042 32015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:46.849059 32010 server_base.cc:1034] running on GCE node
W20260808 18:33:46.849059 32016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:46.849084 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:46.849963 32010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:46.851161 32010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:46.852466 32010 hybrid_clock.cc:648] HybridClock initialized: now 1786214026852504 us; error 96 us; skew 500 ppm
Aug 08 18:33:46 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214026, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:46.855718 32010 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:46.856873 32010 webserver.cc:460] Webserver started at http://127.27.251.254:40241/ using document root <none> and password file <none>
I20260808 18:33:46.857502 32010 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:46.857604 32010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:46.857872 32010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:46.860138 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "85bfdf501d8d40b8a30de0f2a41db00a"
format_stamp: "Formatted at 2026-08-08 18:33:46 on dist-test-slave-fd7l"
server_key: "4f29e1c2dd4dfd48bbea6735c4800d2d"
server_key_iv: "7301e34ad8a980ed20d05865d424c15b"
server_key_version: "encryptionkey@0"
I20260808 18:33:46.860801 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "85bfdf501d8d40b8a30de0f2a41db00a"
format_stamp: "Formatted at 2026-08-08 18:33:46 on dist-test-slave-fd7l"
server_key: "4f29e1c2dd4dfd48bbea6735c4800d2d"
server_key_iv: "7301e34ad8a980ed20d05865d424c15b"
server_key_version: "encryptionkey@0"
I20260808 18:33:46.864945 32010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:33:46.867594 32025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:46.868966 32010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:46.869112 32010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "85bfdf501d8d40b8a30de0f2a41db00a"
format_stamp: "Formatted at 2026-08-08 18:33:46 on dist-test-slave-fd7l"
server_key: "4f29e1c2dd4dfd48bbea6735c4800d2d"
server_key_iv: "7301e34ad8a980ed20d05865d424c15b"
server_key_version: "encryptionkey@0"
I20260808 18:33:46.869247 32010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:46.892195 32010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:46.895675 32010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:46.895888 32010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:46.903708 32010 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:41917
I20260808 18:33:46.904028 32080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:41917 every 8 connection(s)
I20260808 18:33:46.904855 32010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:46.907672 32081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:46.909186 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32010
I20260808 18:33:46.909318 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:46.909623 28655 external_mini_cluster.cc:1428] Setting key 6503cbe8f767d76291c04d1feeaa2707
I20260808 18:33:46.914556 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a: Bootstrap starting.
I20260808 18:33:46.917420 32081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:46.918334 32081 log.cc:824] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:46 dist-test-slave-fd7l krb5kdc[31996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:46.920763 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a: No bootstrap required, opened a new log
I20260808 18:33:46.925099 32081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } }
I20260808 18:33:46.925271 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:46.911085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.27.251.254:41917)
0808 18:33:46.911524 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:46.911544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:46.911588 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:33:46.912400 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:46.912426 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:46.912890 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:46.913257 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:46.913943 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:46.915581 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:46.916490 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:46.917207 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:46.917424 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:46.920371 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:46.920377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:46.920397 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:46.920423 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:46.922223 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:46.922775 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:46.922780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:46.922785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:46.922894 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:46.923232 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:46.923234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:46.923235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:46.923616 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:46.923749 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:46.924099 (+ 350us) server_negotiation.cc:299] Negotiation successful
0808 18:33:46.924428 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260808 18:33:46.925355 32081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:46.925441 32081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bfdf501d8d40b8a30de0f2a41db00a, State: Initialized, Role: FOLLOWER
I20260808 18:33:46.925902 32081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } }
I20260808 18:33:46.926066 32081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:46.926154 32081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:46.926261 32081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:46.927177 32081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } }
I20260808 18:33:46.927558 32081 leader_election.cc:302] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bfdf501d8d40b8a30de0f2a41db00a; no voters:
I20260808 18:33:46.927898 32081 leader_election.cc:288] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:46.928040 32086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:46.928282 32086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [term 1 LEADER]: Becoming Leader. State: Replica: 85bfdf501d8d40b8a30de0f2a41db00a, State: Running, Role: LEADER
I20260808 18:33:46.928663 32086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } }
I20260808 18:33:46.929188 32081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:46.930323 32087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } } }
I20260808 18:33:46.930464 32087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:46.930755 32088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bfdf501d8d40b8a30de0f2a41db00a. Latest consensus state: current_term: 1 leader_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfdf501d8d40b8a30de0f2a41db00a" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41917 } } }
I20260808 18:33:46.930852 32095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:46.930927 32088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:46.934052 32095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:46.939436 32095 catalog_manager.cc:1347] Generated new cluster ID: b3c898baadc647889f5b5c0f328bc4d0
I20260808 18:33:46.939505 32095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:46.956732 32095 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:46.957728 32095 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:46.967060 32095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bfdf501d8d40b8a30de0f2a41db00a: Generated new TSK 0
I20260808 18:33:46.968070 32095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:47.011149 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:47.146267 32110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:47.146529 32110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:47.146576 32110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:47.151069 32110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:47.151124 32110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:47.151180 32110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:47.156234 32110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:47.157220 32110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:47.158174 32110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:47.167723 32117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.167750 32116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.167788 32119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:47.167872 32110 server_base.cc:1034] running on GCE node
I20260808 18:33:47.168326 32110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:47.168988 32110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:47.170171 32110 hybrid_clock.cc:648] HybridClock initialized: now 1786214027170138 us; error 44 us; skew 500 ppm
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:47.173153 32110 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:47.174151 32110 webserver.cc:460] Webserver started at http://127.27.251.193:33583/ using document root <none> and password file <none>
I20260808 18:33:47.174710 32110 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:47.174772 32110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:47.175047 32110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:47.177186 32110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "93d75d3290384fd48ea77e968c6a3e9b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "dae6d83e719cc85ed0dd365289007316"
server_key_iv: "4859adf3c0aa38774d1dc3c94b7e7b92"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.177749 32110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "93d75d3290384fd48ea77e968c6a3e9b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "dae6d83e719cc85ed0dd365289007316"
server_key_iv: "4859adf3c0aa38774d1dc3c94b7e7b92"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.181526 32110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:33:47.183938 32128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.185220 32110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:33:47.185348 32110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "93d75d3290384fd48ea77e968c6a3e9b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "dae6d83e719cc85ed0dd365289007316"
server_key_iv: "4859adf3c0aa38774d1dc3c94b7e7b92"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.185465 32110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:47.196271 32110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:47.199363 32110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:47.199560 32110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:47.200162 32110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:47.201062 32110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:47.201109 32110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.201149 32110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:47.201198 32110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.210379 32110 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:46751
I20260808 18:33:47.210409 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:46751 every 8 connection(s)
I20260808 18:33:47.211256 32110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:47.218598 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32110
I20260808 18:33:47.218751 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:47.219082 28655 external_mini_cluster.cc:1428] Setting key f0ccf2145bb6e274faf71c78a32a593c
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:47.226048 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.213394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:41359 (local address 127.27.251.254:41917)
0808 18:33:47.213545 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:47.213550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:47.214355 (+ 805us) server_negotiation.cc:407] Connection header received
0808 18:33:47.215775 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:47.215779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:47.215841 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:47.215924 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:47.217194 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.217755 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.219254 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.219498 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.219640 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.222329 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:47.222332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:47.222335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:47.222359 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:47.223898 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.224424 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.224428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.224430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.224556 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.224912 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.224917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.224919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.225110 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:47.225220 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:47.225727 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 18:33:47.225867 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:33:47.226890 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.213730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41917 (local address 127.27.251.193:41359)
0808 18:33:47.214203 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:47.214246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:47.215552 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:47.216105 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:47.216112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:47.216480 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:47.217031 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.217045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.217886 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.217890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.219109 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.219133 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.219686 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.219690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.219883 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.220623 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:47.220641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:47.222140 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:47.224033 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.224039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.224055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.224314 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.224661 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.224664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.224666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.224811 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.225221 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:47.225229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:47.225481 (+ 252us) client_negotiation.cc:769] Sending connection context
0808 18:33:47.225682 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 18:33:47.226009 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260808 18:33:47.228225 32257 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41917
I20260808 18:33:47.228525 32257 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:47.229205 32257 heartbeater.cc:511] Master 127.27.251.254:41917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:47.230913 32044 ts_manager.cc:194] Registered new tserver with Master: 93d75d3290384fd48ea77e968c6a3e9b (127.27.251.193:46751)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:47.232236 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:41359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:47.247800 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:47.364140 32266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:47.364387 32266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:47.364431 32266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:47.368487 32266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:47.368618 32266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:47.368680 32266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:47.374190 32266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:47.375265 32266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:47.376271 32266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:47.387120 32274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.387112 32272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.387142 32271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:47.387511 32266 server_base.cc:1034] running on GCE node
I20260808 18:33:47.387954 32266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:47.388803 32266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:47.389966 32266 hybrid_clock.cc:648] HybridClock initialized: now 1786214027389948 us; error 30 us; skew 500 ppm
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:47.392849 32266 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:47.393769 32266 webserver.cc:460] Webserver started at http://127.27.251.194:40719/ using document root <none> and password file <none>
I20260808 18:33:47.394425 32266 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:47.394488 32266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:47.394739 32266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:47.396919 32266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "c4dd130363f3480b99931727aed49157"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "32cce55ec59fa66601ea7e003a1045da"
server_key_iv: "f96ef173b0a85f71dfe960ff1fbb882b"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.397516 32266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "c4dd130363f3480b99931727aed49157"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "32cce55ec59fa66601ea7e003a1045da"
server_key_iv: "f96ef173b0a85f71dfe960ff1fbb882b"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.401607 32266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:33:47.404265 32281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.405680 32266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:33:47.405827 32266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "c4dd130363f3480b99931727aed49157"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "32cce55ec59fa66601ea7e003a1045da"
server_key_iv: "f96ef173b0a85f71dfe960ff1fbb882b"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.405947 32266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:47.439566 32266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:47.442715 32266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:47.442935 32266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:47.443586 32266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:47.444443 32266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:47.444517 32266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.444643 32266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:47.444689 32266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.453809 32266 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:34371
I20260808 18:33:47.453860 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:34371 every 8 connection(s)
I20260808 18:33:47.454741 32266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:47.455374 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32266
I20260808 18:33:47.455526 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:47.455840 28655 external_mini_cluster.cc:1428] Setting key 18e6cf74efb58c4c2bc0542a103a6ff0
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:33:47.471339 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.456765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:43427 (local address 127.27.251.254:41917)
0808 18:33:47.456906 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:47.456911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:47.457950 (+ 1039us) server_negotiation.cc:407] Connection header received
0808 18:33:47.459400 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:47.459403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:47.459459 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:47.459547 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:47.460872 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.461421 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.462956 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.463404 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.463557 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.467084 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:47.467089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:47.467092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:47.467123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:47.469031 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.469665 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.469672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.469674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.469782 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.470165 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.470169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.470171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.470380 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:47.470493 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:47.471010 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 18:33:47.471156 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:33:47.472407 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.457258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41917 (local address 127.27.251.194:43427)
0808 18:33:47.457806 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:47.457848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:47.459161 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:47.459720 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:47.459728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:47.460109 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:47.460723 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.460739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.461558 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.461563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.462810 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.462827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.463613 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.463618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.463862 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.464655 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:47.464681 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:47.466832 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:47.469164 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.469173 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.469194 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.469520 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.469905 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.469910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.469912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.470064 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.470493 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:47.470501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:47.470768 (+ 267us) client_negotiation.cc:769] Sending connection context
0808 18:33:47.470968 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 18:33:47.471318 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":142,"threads_started":1}
I20260808 18:33:47.473754 32412 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41917
I20260808 18:33:47.474002 32412 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:47.474537 32412 heartbeater.cc:511] Master 127.27.251.254:41917 requested a full tablet report, sending...
I20260808 18:33:47.475703 32044 ts_manager.cc:194] Registered new tserver with Master: c4dd130363f3480b99931727aed49157 (127.27.251.194:34371)
I20260808 18:33:47.476644 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:43427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:47.489898 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:47.607038 32418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:47.607291 32418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:47.607352 32418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:47.611541 32418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:47.611609 32418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:47.611667 32418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:47.616945 32418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:47.617955 32418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:47.618963 32418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:47.629029 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.628999 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:47.629022 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:47.629276 32418 server_base.cc:1034] running on GCE node
I20260808 18:33:47.629812 32418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:47.630475 32418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:47.631770 32418 hybrid_clock.cc:648] HybridClock initialized: now 1786214027631753 us; error 40 us; skew 500 ppm
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:47.634792 32418 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:47.635663 32418 webserver.cc:460] Webserver started at http://127.27.251.195:45385/ using document root <none> and password file <none>
I20260808 18:33:47.636245 32418 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:47.636340 32418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:47.636660 32418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:47.638728 32418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "409baa70b083469799b337575edf9d2b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "fe337c76544bb7825a8a75bbedab4823"
server_key_iv: "6646094e0313089fb4eedffd19fb7fba"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.639313 32418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "409baa70b083469799b337575edf9d2b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "fe337c76544bb7825a8a75bbedab4823"
server_key_iv: "6646094e0313089fb4eedffd19fb7fba"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.643687 32418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:33:47.646397 32435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.647706 32418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:47.647951 32418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "409baa70b083469799b337575edf9d2b"
format_stamp: "Formatted at 2026-08-08 18:33:47 on dist-test-slave-fd7l"
server_key: "fe337c76544bb7825a8a75bbedab4823"
server_key_iv: "6646094e0313089fb4eedffd19fb7fba"
server_key_version: "encryptionkey@0"
I20260808 18:33:47.648072 32418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:47.667810 32418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:47.671459 32418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:47.671691 32418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:47.672343 32418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:47.673346 32418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:47.673417 32418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.673494 32418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:47.673555 32418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:47.683212 32418 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:36045
I20260808 18:33:47.683243 32564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:36045 every 8 connection(s)
I20260808 18:33:47.684139 32418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:47.687538 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32418
I20260808 18:33:47.687685 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:47.688067 28655 external_mini_cluster.cc:1428] Setting key d419565c7e619da870a05f91c7816209
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:47.698868 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.686236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:46787 (local address 127.27.251.254:41917)
0808 18:33:47.686384 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:47.686388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:47.687269 (+ 881us) server_negotiation.cc:407] Connection header received
0808 18:33:47.689233 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:47.689237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:47.689292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:47.689404 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:47.690917 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.691531 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.692745 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.692976 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.693060 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.695502 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:47.695505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:47.695508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:47.695533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:47.697069 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.697550 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.697553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.697554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.697638 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.697934 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.697936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.697937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.698106 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:47.698181 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:47.698642 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 18:33:47.698746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:33:47.699690 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.686644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41917 (local address 127.27.251.195:46787)
0808 18:33:47.687105 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:47.687152 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:47.688965 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:47.689625 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:47.689634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:47.690107 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:47.690761 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.690789 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.691647 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.691649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.692618 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:47.692628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.693106 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.693108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:47.693266 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.693802 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:47.693819 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:47.695299 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:47.697170 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.697174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.697186 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.697425 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.697722 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:47.697724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:47.697726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:47.697853 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:47.698195 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:47.698200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:47.698430 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 18:33:47.698582 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 18:33:47.698839 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260808 18:33:47.700910 32565 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41917
I20260808 18:33:47.701171 32565 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:47.701815 32565 heartbeater.cc:511] Master 127.27.251.254:41917 requested a full tablet report, sending...
I20260808 18:33:47.703015 32044 ts_manager.cc:194] Registered new tserver with Master: 409baa70b083469799b337575edf9d2b (127.27.251.195:36045)
I20260808 18:33:47.703603 32044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:46787
I20260808 18:33:47.714202 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[31996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214027, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:47.741068 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:47.732416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.27.251.254:41917)
0808 18:33:47.732640 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:47.732645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:47.732785 (+ 140us) server_negotiation.cc:407] Connection header received
0808 18:33:47.732967 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:47.732970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:47.733014 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:47.733101 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:47.733847 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.734371 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.735278 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:47.735503 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:47.735619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:47.738078 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:47.738088 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:47.738091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:47.738115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:47.739339 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.739776 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.739784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.739785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.739858 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:47.740140 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:47.740144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:47.740146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:47.740365 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:47.740509 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:47.740772 (+ 263us) server_negotiation.cc:299] Negotiation successful
0808 18:33:47.740896 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 18:33:47.741467 32044 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56818
I20260808 18:33:47.742623 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32110
I20260808 18:33:47.752008 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32266
I20260808 18:33:47.762048 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32418
I20260808 18:33:47.769354 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32010
2026-08-08T18:33:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2317 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[32580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[32580](info): set up 2 sockets
Aug 08 18:33:47 dist-test-slave-fd7l krb5kdc[32580](info): commencing operation
krb5kdc: starting...
W20260808 18:33:48.879753 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:48 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:49.013841 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35165
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:49.137286 32594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:49.137635 32594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:49.137722 32594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:49.141858 32594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:49.141958 32594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:49.142001 32594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:49.142062 32594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:49.142107 32594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:49.147523 32594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:49.148779 32594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:49.149928 32594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:49.159102 32599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:49.159153 32604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.159529 32594 server_base.cc:1034] running on GCE node
W20260808 18:33:49.159237 32600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.160158 32594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:49.161435 32594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:49.162622 32594 hybrid_clock.cc:648] HybridClock initialized: now 1786214029162590 us; error 45 us; skew 500 ppm
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:49.165804 32594 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:49.166833 32594 webserver.cc:460] Webserver started at http://127.27.251.254:39309/ using document root <none> and password file <none>
I20260808 18:33:49.167500 32594 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:49.167599 32594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:49.167876 32594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:49.170213 32594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "6cca9481373b4be6885b2861d6260ccf"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "0463465e654705b46224948ac65c4f72"
server_key_iv: "2d12d06c2a57d926a8d0e69a3f631c31"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.170907 32594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "6cca9481373b4be6885b2861d6260ccf"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "0463465e654705b46224948ac65c4f72"
server_key_iv: "2d12d06c2a57d926a8d0e69a3f631c31"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.175009 32594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:49.177737 32612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.179064 32594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:49.179229 32594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "6cca9481373b4be6885b2861d6260ccf"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "0463465e654705b46224948ac65c4f72"
server_key_iv: "2d12d06c2a57d926a8d0e69a3f631c31"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.179353 32594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:49.193610 32594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:49.196723 32594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:49.196930 32594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:49.204790 32594 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35165
I20260808 18:33:49.204826 32674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35165 every 8 connection(s)
I20260808 18:33:49.205896 32594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:49.208716 32676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:49.210399 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32594
I20260808 18:33:49.210515 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:49.210820 28655 external_mini_cluster.cc:1428] Setting key 2e496c744f6d2f9e480ebea0ec766558
I20260808 18:33:49.215168 32676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf: Bootstrap starting.
I20260808 18:33:49.217793 32676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:49.218731 32676 log.cc:824] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:49.220948 32676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf: No bootstrap required, opened a new log
I20260808 18:33:49.224730 32676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } }
I20260808 18:33:49.225004 32676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:49.225090 32676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cca9481373b4be6885b2861d6260ccf, State: Initialized, Role: FOLLOWER
I20260808 18:33:49.225484 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.212215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.27.251.254:35165)
0808 18:33:49.212640 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:49.212656 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:49.212694 (+ 38us) server_negotiation.cc:407] Connection header received
0808 18:33:49.213307 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:49.213323 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:49.213653 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:49.213931 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:49.214872 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.215912 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.217120 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.217617 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.217738 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.220598 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:49.220603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:49.220617 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:49.220650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:49.222357 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.222839 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.222844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.222848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.222960 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.223266 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.223270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.223273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.223676 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:49.223902 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:49.224184 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 18:33:49.224488 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
I20260808 18:33:49.225529 32676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } }
I20260808 18:33:49.225747 32676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:49.225823 32676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:49.225929 32676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:49.227058 32676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } }
I20260808 18:33:49.227401 32676 leader_election.cc:302] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cca9481373b4be6885b2861d6260ccf; no voters:
I20260808 18:33:49.227758 32676 leader_election.cc:288] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:49.227842 32682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:49.228099 32682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [term 1 LEADER]: Becoming Leader. State: Replica: 6cca9481373b4be6885b2861d6260ccf, State: Running, Role: LEADER
I20260808 18:33:49.228469 32682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } }
I20260808 18:33:49.229086 32676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:49.230268 32683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cca9481373b4be6885b2861d6260ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } } }
I20260808 18:33:49.230396 32683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:49.230662 32684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cca9481373b4be6885b2861d6260ccf. Latest consensus state: current_term: 1 leader_uuid: "6cca9481373b4be6885b2861d6260ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca9481373b4be6885b2861d6260ccf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35165 } } }
I20260808 18:33:49.230757 32684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:49.231060 32691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:49.233947 32691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:49.239655 32691 catalog_manager.cc:1347] Generated new cluster ID: 4318cde74969430bab79a4170e6db23b
I20260808 18:33:49.239724 32691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:49.251271 32691 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:49.252236 32691 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:49.259469 32691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cca9481373b4be6885b2861d6260ccf: Generated new TSK 0
I20260808 18:33:49.260191 32691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:49.291805 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:49.408675 32705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:49.408928 32705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:49.408972 32705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:49.412976 32705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:49.413033 32705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:49.413089 32705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:49.418397 32705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:49.419375 32705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:49.420360 32705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:49.430321 32711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:49.430279 32710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.430372 32705 server_base.cc:1034] running on GCE node
W20260808 18:33:49.430279 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.430987 32705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:49.431707 32705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:49.432898 32705 hybrid_clock.cc:648] HybridClock initialized: now 1786214029432880 us; error 28 us; skew 500 ppm
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:49.435694 32705 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:49.436607 32705 webserver.cc:460] Webserver started at http://127.27.251.193:42665/ using document root <none> and password file <none>
I20260808 18:33:49.437163 32705 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:49.437222 32705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:49.437428 32705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:49.439453 32705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "2fa7a542156f430d8742c82ee0a729e5"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "24ef7c568ad3a6b62eb78878e266ef9d"
server_key_iv: "0ce3a8fb3649446da116ea572164292e"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.439966 32705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "2fa7a542156f430d8742c82ee0a729e5"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "24ef7c568ad3a6b62eb78878e266ef9d"
server_key_iv: "0ce3a8fb3649446da116ea572164292e"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.443696 32705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:49.446226 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.447487 32705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:49.447589 32705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "2fa7a542156f430d8742c82ee0a729e5"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "24ef7c568ad3a6b62eb78878e266ef9d"
server_key_iv: "0ce3a8fb3649446da116ea572164292e"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.447688 32705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:49.479465 32705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:49.482633 32705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:49.482872 32705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:49.483536 32705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:49.484558 32705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:49.484616 32705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.484665 32705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:49.484687 32705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.494065 32705 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:40273
I20260808 18:33:49.494086 385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:40273 every 8 connection(s)
I20260808 18:33:49.494985 32705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:49.499253 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 32705
I20260808 18:33:49.499413 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:49.499694 28655 external_mini_cluster.cc:1428] Setting key 0ec5567ca0f98c9c049da252c84cc5b7
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:49.510154 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.496836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:51911 (local address 127.27.251.254:35165)
0808 18:33:49.496991 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:49.496995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:49.497881 (+ 886us) server_negotiation.cc:407] Connection header received
0808 18:33:49.499329 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:49.499333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:49.499391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:49.499500 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:49.500839 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.501764 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.502966 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.503249 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.503377 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.506025 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:49.506038 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:49.506044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:49.506080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:49.507858 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.508398 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.508404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.508408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.508501 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.508879 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.508884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.508886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.509153 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:49.509287 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:49.509780 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 18:33:49.509914 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:33:49.510960 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.497275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35165 (local address 127.27.251.193:51911)
0808 18:33:49.497728 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:49.497770 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:49.499119 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:49.499691 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:49.499698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:49.500109 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:49.500687 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.500702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.501913 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.501917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.502832 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.502842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.503445 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.503449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.503612 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.504271 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:49.504288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:49.505842 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:49.507999 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.508004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.508016 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.508292 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.508624 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.508628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.508630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.508772 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.509282 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:49.509287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:49.509528 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 18:33:49.509687 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 18:33:49.509948 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260808 18:33:49.512279 386 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35165
I20260808 18:33:49.512494 386 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:49.513036 386 heartbeater.cc:511] Master 127.27.251.254:35165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:49.514850 32631 ts_manager.cc:194] Registered new tserver with Master: 2fa7a542156f430d8742c82ee0a729e5 (127.27.251.193:40273)
I20260808 18:33:49.516234 32631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:51911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:49.531371 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:49.654443 392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:49.654752 392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:49.654798 392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:49.659391 392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:49.659463 392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:49.659565 392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:49.665019 392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:49.666126 392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:49.667176 392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:49.677297 400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:49.677307 397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.677561 392 server_base.cc:1034] running on GCE node
W20260808 18:33:49.677310 398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.678160 392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:49.678859 392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:49.680027 392 hybrid_clock.cc:648] HybridClock initialized: now 1786214029679996 us; error 40 us; skew 500 ppm
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:49.683151 392 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:49.684180 392 webserver.cc:460] Webserver started at http://127.27.251.194:38415/ using document root <none> and password file <none>
I20260808 18:33:49.684836 392 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:49.684899 392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:49.685125 392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:49.687192 392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "950884b297e844d1b1ae23daaed18fb0"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "7ae2f146ed2b66693f6744a6ec5442b7"
server_key_iv: "319c0eda5072d8199bdb2e948897fd68"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.687762 392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "950884b297e844d1b1ae23daaed18fb0"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "7ae2f146ed2b66693f6744a6ec5442b7"
server_key_iv: "319c0eda5072d8199bdb2e948897fd68"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.691592 392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:49.694178 410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.695439 392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:49.695611 392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "950884b297e844d1b1ae23daaed18fb0"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "7ae2f146ed2b66693f6744a6ec5442b7"
server_key_iv: "319c0eda5072d8199bdb2e948897fd68"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.695709 392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:49.711205 392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:49.714184 392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:49.714387 392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:49.715001 392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:49.715843 392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:49.715891 392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.715961 392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:49.716005 392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.725459 392 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:40827
I20260808 18:33:49.725534 536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:40827 every 8 connection(s)
I20260808 18:33:49.726411 392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:49.728886 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 392
I20260808 18:33:49.729013 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:49.729328 28655 external_mini_cluster.cc:1428] Setting key 50c8db6cc7014c43154d6e8cc67e689d
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:33:49.742537 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.728252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:38165 (local address 127.27.251.254:35165)
0808 18:33:49.728387 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:49.728391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:49.729377 (+ 986us) server_negotiation.cc:407] Connection header received
0808 18:33:49.731372 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:49.731377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:49.731429 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:49.731547 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:49.732819 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.733536 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.735366 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.735589 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.735680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.738800 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:49.738803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:49.738805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:49.738830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:49.740312 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.740899 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.740902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.740906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.740986 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.741369 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.741373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.741374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.741581 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:49.741692 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:49.742232 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 18:33:49.742354 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:33:49.743400 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.728711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35165 (local address 127.27.251.194:38165)
0808 18:33:49.729190 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:49.729266 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:49.731147 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:49.731699 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:49.731705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:49.732063 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:49.732652 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.732673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.733672 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.733679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.735199 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.735214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.735756 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.735763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.736034 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.736841 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:49.736863 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:49.738617 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:49.740437 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.740443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.740457 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.740787 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.741103 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.741106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.741108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.741256 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.741692 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:49.741710 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:49.741953 (+ 243us) client_negotiation.cc:769] Sending connection context
0808 18:33:49.742172 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 18:33:49.742495 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":142,"threads_started":1}
I20260808 18:33:49.744678 537 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35165
I20260808 18:33:49.744935 537 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:49.745605 537 heartbeater.cc:511] Master 127.27.251.254:35165 requested a full tablet report, sending...
I20260808 18:33:49.746821 32631 ts_manager.cc:194] Registered new tserver with Master: 950884b297e844d1b1ae23daaed18fb0 (127.27.251.194:40827)
I20260808 18:33:49.747759 32631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:38165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:49.760744 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:49.876646 542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:49.876909 542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:49.876955 542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:49.881011 542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:49.881062 542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:49.881142 542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:49.886534 542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:49.887557 542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:49.888623 542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:49.898444 550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:49.898444 549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.898645 542 server_base.cc:1034] running on GCE node
W20260808 18:33:49.898469 552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:49.899192 542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:49.899971 542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:49.901189 542 hybrid_clock.cc:648] HybridClock initialized: now 1786214029901172 us; error 28 us; skew 500 ppm
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:49.904002 542 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:49.904954 542 webserver.cc:460] Webserver started at http://127.27.251.195:35133/ using document root <none> and password file <none>
I20260808 18:33:49.905545 542 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:49.905642 542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:49.905879 542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:49.908015 542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "d6c9cfc6b8934de6a84f6917819f8302"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "9bacbdee92008b4e3b8ddef7c33f5964"
server_key_iv: "07a9ef640f6fd0519b68ebad53a83eef"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.908655 542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "d6c9cfc6b8934de6a84f6917819f8302"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "9bacbdee92008b4e3b8ddef7c33f5964"
server_key_iv: "07a9ef640f6fd0519b68ebad53a83eef"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.912420 542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:49.914809 560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.916042 542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:49.916179 542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "d6c9cfc6b8934de6a84f6917819f8302"
format_stamp: "Formatted at 2026-08-08 18:33:49 on dist-test-slave-fd7l"
server_key: "9bacbdee92008b4e3b8ddef7c33f5964"
server_key_iv: "07a9ef640f6fd0519b68ebad53a83eef"
server_key_version: "encryptionkey@0"
I20260808 18:33:49.916289 542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:49.945724 542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:49.949290 542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:49.949529 542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:49.950205 542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:49.951167 542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:49.951242 542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.951319 542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:49.951372 542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:49.961127 542 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:41993
I20260808 18:33:49.961148 686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:41993 every 8 connection(s)
I20260808 18:33:49.962133 542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:49.968360 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 542
I20260808 18:33:49.968487 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:49.968823 28655 external_mini_cluster.cc:1428] Setting key b18697c4b82aa16411a7f4dde915734e
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:49.977054 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.964135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:51017 (local address 127.27.251.254:35165)
0808 18:33:49.964290 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:49.964294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:49.965337 (+ 1043us) server_negotiation.cc:407] Connection header received
0808 18:33:49.966754 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:49.966757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:49.966806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:49.966888 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:49.968168 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.968998 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.970700 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.971030 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.971152 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.973688 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:49.973691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:49.973693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:49.973716 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:49.975040 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.975567 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.975569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.975570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.975651 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:49.976013 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:49.976015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:49.976016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:49.976221 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:49.976301 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:49.976833 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 18:33:49.976933 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:33:49.977962 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:49.964678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35165 (local address 127.27.251.195:51017)
0808 18:33:49.965175 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:49.965220 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:49.966518 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:49.967066 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:49.967074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:49.967467 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:49.967987 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.968001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.969193 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.969197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.970568 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:49.970579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:49.971193 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:49.971196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:49.971344 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:49.971979 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:49.971996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:49.973532 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:49.975185 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.975189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.975202 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.975459 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.975736 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:49.975738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:49.975740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:49.975908 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:49.976315 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:49.976320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:49.976616 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 18:33:49.976785 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 18:33:49.977044 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":146,"threads_started":1}
I20260808 18:33:49.979169 687 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35165
I20260808 18:33:49.979418 687 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:49.979911 687 heartbeater.cc:511] Master 127.27.251.254:35165 requested a full tablet report, sending...
I20260808 18:33:49.980976 32631 ts_manager.cc:194] Registered new tserver with Master: d6c9cfc6b8934de6a84f6917819f8302 (127.27.251.195:41993)
I20260808 18:33:49.981551 32631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:51017
I20260808 18:33:49.982847 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:33:49 dist-test-slave-fd7l krb5kdc[32580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:33:50 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:50.010797 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:50.001021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.27.251.193:40273)
0808 18:33:50.001393 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:50.001413 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:50.001437 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:33:50.001491 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:50.001496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:50.001686 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:50.001811 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:50.002529 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:50.003320 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:50.004145 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:50.004567 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:50.004665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:50.007120 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:50.007131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:50.007145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:50.007170 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:50.008910 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:50.009369 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:50.009375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:50.009376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:50.009457 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:50.009775 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:50.009779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:50.009781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:50.010032 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:50.010286 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:50.010448 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 18:33:50.010606 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":160,"threads_started":1}
Aug 08 18:33:50 dist-test-slave-fd7l krb5kdc[32580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:50.022550 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:50.013820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.27.251.254:35165)
0808 18:33:50.013983 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:50.013989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:50.014003 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:33:50.014290 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:50.014294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:50.014355 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:50.014461 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:50.015108 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:50.015657 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:50.016551 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:50.016794 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:50.016932 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:50.019270 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:50.019273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:50.019275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:50.019298 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:50.020929 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:50.021390 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:50.021392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:50.021393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:50.021469 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:50.021776 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:50.021778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:50.021779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:50.021993 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:50.022104 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:50.022317 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 18:33:50.022418 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 18:33:50.023000 32631 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58836
I20260808 18:33:50.023723 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32705
I20260808 18:33:50.031703 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 392
I20260808 18:33:50.041958 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 542
I20260808 18:33:50.049599 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 32594
2026-08-08T18:33:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2279 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:50 dist-test-slave-fd7l krb5kdc[700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:50 dist-test-slave-fd7l krb5kdc[700](info): set up 2 sockets
Aug 08 18:33:50 dist-test-slave-fd7l krb5kdc[700](info): commencing operation
krb5kdc: starting...
W20260808 18:33:51.180716 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 18:33:51.436457 28655 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.154s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/test-user.keytab.
W20260808 18:33:51.985101 28655 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.548s user 0.000s sys 0.002s
Aug 08 18:33:51 dist-test-slave-fd7l krb5kdc[700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
W20260808 18:33:52.760881 28655 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.251.254: real 0.749s user 0.000s sys 0.004s
I20260808 18:33:52.761439 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:44071
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:44071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:52.879346 714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:52.879616 714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:52.879659 714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:52.883695 714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:52.883754 714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:52.883773 714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:52.883790 714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:52.883805 714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:52.889213 714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:44071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:44071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:52.890293 714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:52.891296 714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:52.900317 719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:52.900319 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:52.900377 720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:52.900722 714 server_base.cc:1034] running on GCE node
I20260808 18:33:52.901471 714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:52.902733 714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:52.903939 714 hybrid_clock.cc:648] HybridClock initialized: now 1786214032903886 us; error 65 us; skew 500 ppm
Aug 08 18:33:52 dist-test-slave-fd7l krb5kdc[700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214032, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:52.907007 714 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:52.907824 714 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:52.907925 714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:52.908179 714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:52.910529 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "a120d115e71c47b89a4cb09645385220"
format_stamp: "Formatted at 2026-08-08 18:33:52 on dist-test-slave-fd7l"
server_key: "ca1479e1c55611613817dafc2a944f54"
server_key_iv: "676227bc024298692c5e1ad2fb2e5fca"
server_key_version: "encryptionkey@0"
I20260808 18:33:52.911176 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "a120d115e71c47b89a4cb09645385220"
format_stamp: "Formatted at 2026-08-08 18:33:52 on dist-test-slave-fd7l"
server_key: "ca1479e1c55611613817dafc2a944f54"
server_key_iv: "676227bc024298692c5e1ad2fb2e5fca"
server_key_version: "encryptionkey@0"
I20260808 18:33:52.915424 714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:33:52.918010 728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:52.919312 714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:52.919469 714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a120d115e71c47b89a4cb09645385220"
format_stamp: "Formatted at 2026-08-08 18:33:52 on dist-test-slave-fd7l"
server_key: "ca1479e1c55611613817dafc2a944f54"
server_key_iv: "676227bc024298692c5e1ad2fb2e5fca"
server_key_version: "encryptionkey@0"
I20260808 18:33:52.919589 714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:52.940694 714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:52.944134 714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:52.944363 714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:52.953051 714 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:44071
I20260808 18:33:52.953352 786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:44071 every 8 connection(s)
I20260808 18:33:52.953838 714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:52.956812 787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:52.959493 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 714
I20260808 18:33:52.959599 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:52.959909 28655 external_mini_cluster.cc:1428] Setting key e03e53cbef7c3b4b123df0d600be657e
I20260808 18:33:52.963625 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: Bootstrap starting.
I20260808 18:33:52.966603 787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:52.967494 787 log.cc:824] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:52.969743 787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: No bootstrap required, opened a new log
Aug 08 18:33:52 dist-test-slave-fd7l krb5kdc[700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:52.973443 787 raft_consensus.cc:348] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } }
I20260808 18:33:52.973706 787 raft_consensus.cc:374] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:52.973778 787 raft_consensus.cc:729] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a120d115e71c47b89a4cb09645385220, State: Initialized, Role: FOLLOWER
I20260808 18:33:52.974308 787 consensus_queue.cc:260] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } }
I20260808 18:33:52.974484 787 raft_consensus.cc:388] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:52.974570 787 raft_consensus.cc:482] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:52.974720 787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:52.975865 787 raft_consensus.cc:504] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } }
I20260808 18:33:52.976266 787 leader_election.cc:302] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a120d115e71c47b89a4cb09645385220; no voters:
I20260808 18:33:52.976725 787 leader_election.cc:288] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:52.976804 793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:52.977063 793 raft_consensus.cc:686] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [term 1 LEADER]: Becoming Leader. State: Replica: a120d115e71c47b89a4cb09645385220, State: Running, Role: LEADER
I20260808 18:33:52.977442 793 consensus_queue.cc:237] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } }
I20260808 18:33:52.977773 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:52.961591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.27.251.254:44071)
0808 18:33:52.962134 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:52.962173 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:52.962221 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:33:52.963068 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:52.963092 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:52.963523 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:52.963929 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:52.964850 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:52.966399 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:52.967353 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:52.968066 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:52.968193 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:52.971397 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:52.971402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:52.971416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:52.971446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:52.974043 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:52.974631 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:52.974638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:52.974644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:52.974845 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:52.975195 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:52.975198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:52.975199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:52.975689 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:52.975881 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:52.976222 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 18:33:52.976606 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":148,"threads_started":1}
I20260808 18:33:52.978149 787 sys_catalog.cc:564] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:52.979393 795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a120d115e71c47b89a4cb09645385220. Latest consensus state: current_term: 1 leader_uuid: "a120d115e71c47b89a4cb09645385220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } } }
I20260808 18:33:52.979518 795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:52.979812 794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a120d115e71c47b89a4cb09645385220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a120d115e71c47b89a4cb09645385220" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44071 } } }
I20260808 18:33:52.979887 794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:52.980145 802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:52.983458 802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:52.989662 802 catalog_manager.cc:1347] Generated new cluster ID: 939332db759649a49e03f559860efc30
I20260808 18:33:52.989744 802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:53.009550 802 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:53.010921 802 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:53.022022 802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: Generated new TSK 0
I20260808 18:33:53.022984 802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:53.056463 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:53.178485 815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:53.178810 815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:53.178895 815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:53.183298 815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:53.183406 815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:53.183706 815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:53.190303 815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:53.191515 815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:53.192616 815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:53.202674 823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.202656 820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.202694 821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:53.202765 815 server_base.cc:1034] running on GCE node
I20260808 18:33:53.203364 815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:53.204103 815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:53.205299 815 hybrid_clock.cc:648] HybridClock initialized: now 1786214033205280 us; error 29 us; skew 500 ppm
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:53.208164 815 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:53.209028 815 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:53.209124 815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:53.209374 815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:53.211496 815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "c786a553e8964e3791991bd2d3dc0316"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "02c1cef1abf30fe9ef384a69fa6fdcf5"
server_key_iv: "d12d6eb22d3adb25cea154a4b4cb96d8"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.212082 815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "c786a553e8964e3791991bd2d3dc0316"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "02c1cef1abf30fe9ef384a69fa6fdcf5"
server_key_iv: "d12d6eb22d3adb25cea154a4b4cb96d8"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.215900 815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:53.218628 830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.220309 815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:53.220438 815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c786a553e8964e3791991bd2d3dc0316"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "02c1cef1abf30fe9ef384a69fa6fdcf5"
server_key_iv: "d12d6eb22d3adb25cea154a4b4cb96d8"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.220607 815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:53.238441 815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:53.242040 815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:53.242262 815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:53.242908 815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:53.243794 815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:53.243880 815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.243980 815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:53.244035 815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.253336 815 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:32957
I20260808 18:33:53.253353 958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:32957 every 8 connection(s)
I20260808 18:33:53.253979 815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:53.254971 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 815
I20260808 18:33:53.255105 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:53.255436 28655 external_mini_cluster.cc:1428] Setting key 28ebe4db81d925c3c5126043d045f6df
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:53.270781 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.256117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56487 (local address 127.27.251.254:44071)
0808 18:33:53.256350 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.256355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.257417 (+ 1062us) server_negotiation.cc:407] Connection header received
0808 18:33:53.258899 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.258902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.258958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.259040 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:53.260276 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.260899 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.262770 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.263007 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.263175 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.266405 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:53.266408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:53.266410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:53.266439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:53.268322 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.268989 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.268992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.268993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.269084 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.269427 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.269429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.269430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.269686 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:53.269768 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.270368 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.270536 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260808 18:33:53.271713 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.256662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44071 (local address 127.27.251.193:56487)
0808 18:33:53.257271 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.257318 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.258640 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.259225 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.259232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.259603 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:53.260114 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.260138 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.261036 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.261042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.262619 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.262636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.263252 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.263258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.263504 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.264175 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:53.264205 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:53.266175 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:53.268445 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.268450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.268467 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.268888 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.269191 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.269193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.269194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.269340 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.269808 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:53.269814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:53.270097 (+ 283us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.270258 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.270645 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":184,"threads_started":1}
I20260808 18:33:53.273257 959 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44071
I20260808 18:33:53.273519 959 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:53.274029 959 heartbeater.cc:511] Master 127.27.251.254:44071 requested a full tablet report, sending...
I20260808 18:33:53.275701 737 ts_manager.cc:194] Registered new tserver with Master: c786a553e8964e3791991bd2d3dc0316 (127.27.251.193:32957)
I20260808 18:33:53.277232 737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:56487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:53.287773 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:53.407778 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:53.408048 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:53.408097 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:53.412117 966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:53.412182 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:53.412245 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:53.417668 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:53.418704 966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:53.419719 966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:53.429792 974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.429850 971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.429850 972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:53.430159 966 server_base.cc:1034] running on GCE node
I20260808 18:33:53.430516 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:53.431164 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:53.432368 966 hybrid_clock.cc:648] HybridClock initialized: now 1786214033432351 us; error 29 us; skew 500 ppm
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:53.435307 966 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:53.436085 966 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:53.436182 966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:53.436424 966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:53.438510 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "254e279ee4c4494b833f8582f29df43b"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "31d0c58171cdab147cb5e3b2de7dc6b1"
server_key_iv: "84515e3a5a501f0bdc14f760ab46421a"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.439093 966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "254e279ee4c4494b833f8582f29df43b"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "31d0c58171cdab147cb5e3b2de7dc6b1"
server_key_iv: "84515e3a5a501f0bdc14f760ab46421a"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.442893 966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:53.445422 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.446715 966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:53.446870 966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "254e279ee4c4494b833f8582f29df43b"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "31d0c58171cdab147cb5e3b2de7dc6b1"
server_key_iv: "84515e3a5a501f0bdc14f760ab46421a"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.446991 966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:53.459159 966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:53.462759 966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:53.462966 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:53.463619 966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:53.464859 966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:53.464916 966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.464974 966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:53.465006 966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.476109 966 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:41365
I20260808 18:33:53.476130 1105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:41365 every 8 connection(s)
I20260808 18:33:53.476831 966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:53.485566 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 966
I20260808 18:33:53.485714 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:53.486012 28655 external_mini_cluster.cc:1428] Setting key 1bfaefab5be7813e569fc998f457ec9b
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:53.491876 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.478888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:41005 (local address 127.27.251.254:44071)
0808 18:33:53.479039 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.479043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.480225 (+ 1182us) server_negotiation.cc:407] Connection header received
0808 18:33:53.481726 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.481730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.481795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.481903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:53.483095 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.483644 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.484888 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.485126 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.485243 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.488056 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:53.488059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:53.488061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:53.488084 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:53.489665 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.490162 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.490168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.490172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.490274 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.490602 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.490607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.490610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.490862 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:53.490976 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.491471 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.491632 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:53.492609 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.479432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44071 (local address 127.27.251.194:41005)
0808 18:33:53.480062 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.480107 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.481470 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.482080 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.482086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.482457 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:53.482939 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.482951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.483786 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.483791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.484758 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.484768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.485291 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.485295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.485476 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.486133 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:53.486159 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:53.487846 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:53.489792 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.489797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.489809 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.490062 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.490371 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.490373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.490374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.490509 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.490973 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:53.490978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:53.491214 (+ 236us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.491370 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.491655 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":113,"threads_started":1}
I20260808 18:33:53.493846 1106 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44071
I20260808 18:33:53.494064 1106 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:53.494493 1106 heartbeater.cc:511] Master 127.27.251.254:44071 requested a full tablet report, sending...
I20260808 18:33:53.495496 737 ts_manager.cc:194] Registered new tserver with Master: 254e279ee4c4494b833f8582f29df43b (127.27.251.194:41365)
I20260808 18:33:53.496025 737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:41005
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:53.515782 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:53.631464 1111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:53.631764 1111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:53.631852 1111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:53.635818 1111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:53.635907 1111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:53.636024 1111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:53.641252 1111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:53.642320 1111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:53.643376 1111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:53.653028 1111 server_base.cc:1034] running on GCE node
W20260808 18:33:53.653043 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.653043 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:53.653198 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:53.653610 1111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:53.654378 1111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:53.655581 1111 hybrid_clock.cc:648] HybridClock initialized: now 1786214033655565 us; error 34 us; skew 500 ppm
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:53.658452 1111 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:53.659273 1111 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:53.659369 1111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:53.659614 1111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:53.661732 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "72b55d5537a479d3ab1a0c44253801c0"
server_key_iv: "077732e89d916ce6df0340ce2e3b1517"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.662329 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "72b55d5537a479d3ab1a0c44253801c0"
server_key_iv: "077732e89d916ce6df0340ce2e3b1517"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.666200 1111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:53.668699 1125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.669965 1111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:53.670123 1111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88"
format_stamp: "Formatted at 2026-08-08 18:33:53 on dist-test-slave-fd7l"
server_key: "72b55d5537a479d3ab1a0c44253801c0"
server_key_iv: "077732e89d916ce6df0340ce2e3b1517"
server_key_version: "encryptionkey@0"
I20260808 18:33:53.670239 1111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:53.691107 1111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:53.694176 1111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:53.694388 1111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:53.695043 1111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:53.695931 1111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:53.696003 1111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.696074 1111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:53.696116 1111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:53.706164 1111 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:39111
I20260808 18:33:53.706180 1247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:39111 every 8 connection(s)
I20260808 18:33:53.706799 1111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:53.712836 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1111
I20260808 18:33:53.712996 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:53.713312 28655 external_mini_cluster.cc:1428] Setting key 589f777f1d8e53f98130266e0f122bea
Aug 08 18:33:53 dist-test-slave-fd7l krb5kdc[700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:53.721015 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.708723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:52907 (local address 127.27.251.254:44071)
0808 18:33:53.708906 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.708911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.709839 (+ 928us) server_negotiation.cc:407] Connection header received
0808 18:33:53.711163 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.711166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.711216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.711291 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:53.712491 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.713233 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.714677 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.714973 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.715091 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.717707 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:53.717710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:53.717712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:53.717736 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:53.719088 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.719554 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.719557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.719558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.719637 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.719960 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.719963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.719965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.720159 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:53.720245 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.720794 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.720895 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:33:53.721817 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.709172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44071 (local address 127.27.251.195:52907)
0808 18:33:53.709663 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.709714 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.710961 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.711470 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.711477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.711834 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:53.712351 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.712365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.713367 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.713376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.714552 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.714563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.715123 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.715139 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.715298 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.716000 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:53.716018 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:53.717545 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:53.719197 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.719201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.719212 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.719462 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.719721 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:53.719723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:53.719724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:53.719856 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:53.720256 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:53.720261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:53.720501 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.720735 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.720987 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260808 18:33:53.722901 1248 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44071
I20260808 18:33:53.723165 1248 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:53.723717 1248 heartbeater.cc:511] Master 127.27.251.254:44071 requested a full tablet report, sending...
I20260808 18:33:53.724803 737 ts_manager.cc:194] Registered new tserver with Master: e7a486a1773545b5bfe6d3e9e0ff2c88 (127.27.251.195:39111)
I20260808 18:33:53.725400 737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:52907
I20260808 18:33:53.727341 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:33:53.736868 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.729439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.27.251.254:44071)
0808 18:33:53.729584 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.729591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.729807 (+ 216us) server_negotiation.cc:407] Connection header received
0808 18:33:53.729966 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.729969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.730015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.730100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:53.730817 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.731349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.732197 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.732450 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.732584 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.733918 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:53.733922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:53.733925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:53.733950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:53.735115 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.735556 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.735561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.735562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.735652 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:53.735936 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:53.735939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:53.735941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:53.736166 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:53.736309 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.736576 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.736696 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:53.739544 737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:53.742734 737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:53.758299 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.752727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39111 (local address 127.0.0.1:43438)
0808 18:33:53.752957 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.752972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.753051 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.753973 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.753976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.753993 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:53.754087 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.754121 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755547 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.755550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.756691 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.756703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.757719 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.757723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.757831 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.757875 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.757995 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.758132 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":115,"thread_start_us":91,"threads_started":1}
I20260808 18:33:53.758788 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.751612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:32957 (local address 127.0.0.1:36382)
0808 18:33:53.752677 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.752692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.752834 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.753609 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.753615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.753639 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:53.753876 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.753885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755443 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.755446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.757159 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.757166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.758322 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.758328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.758451 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.758475 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.758547 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.758620 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":235,"thread_start_us":100,"threads_started":1}
I20260808 18:33:53.758981 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.753158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.27.251.195:39111)
0808 18:33:53.753567 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.753579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.753611 (+ 32us) server_negotiation.cc:407] Connection header received
0808 18:33:53.753679 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.753682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.753804 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.753968 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:53.754245 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755385 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.756817 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.757586 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.757710 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.757840 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.758043 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.758770 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":169,"threads_started":1}
I20260808 18:33:53.759296 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.752101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.27.251.193:32957)
0808 18:33:53.752462 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.752469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.753001 (+ 532us) server_negotiation.cc:407] Connection header received
0808 18:33:53.753097 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.753104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.753280 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.753440 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:53.754019 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755293 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.757306 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.758010 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.758119 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.758253 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.758975 (+ 722us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.759114 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":134,"threads_started":1}
I20260808 18:33:53.759606 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.751694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.27.251.194:41365)
0808 18:33:53.752121 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.752132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.752779 (+ 647us) server_negotiation.cc:407] Connection header received
0808 18:33:53.753031 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.753037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.753261 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.753402 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:53.754018 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755256 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.757294 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.758124 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.758248 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.758696 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.759319 (+ 623us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.759445 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":120,"threads_started":1}
I20260808 18:33:53.759652 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.751598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:41365 (local address 127.0.0.1:34974)
0808 18:33:53.752636 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.752691 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.752863 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.753609 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.753615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.753653 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:53.753876 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.753885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.755443 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.755446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.757137 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.757159 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.758765 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.758768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.758971 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.759008 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.759279 (+ 271us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.759522 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":291,"spinlock_wait_cycles":9216,"thread_start_us":98,"threads_started":1}
I20260808 18:33:53.762028 885 tablet_service.cc:1467] Processing CreateTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 (DEFAULT_TABLE table=test-table [id=cc0e728b88bc4786ac681e0751fe0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:53.762099 1180 tablet_service.cc:1467] Processing CreateTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 (DEFAULT_TABLE table=test-table [id=cc0e728b88bc4786ac681e0751fe0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:53.762578 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 (DEFAULT_TABLE table=test-table [id=cc0e728b88bc4786ac681e0751fe0fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:53.763229 1180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da8f75d7a384465b8e95d5fab0ecb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:53.763417 885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da8f75d7a384465b8e95d5fab0ecb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:53.763530 1026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da8f75d7a384465b8e95d5fab0ecb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:53.770794 1268 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: Bootstrap starting.
I20260808 18:33:53.771095 1269 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Bootstrap starting.
I20260808 18:33:53.771445 1270 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Bootstrap starting.
I20260808 18:33:53.773041 1268 tablet_bootstrap.cc:654] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:53.773294 1269 tablet_bootstrap.cc:654] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:53.773351 1270 tablet_bootstrap.cc:654] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:53.773919 1268 log.cc:824] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:53.774748 1269 log.cc:824] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:53.774748 1270 log.cc:824] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:53.776686 1270 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: No bootstrap required, opened a new log
I20260808 18:33:53.776693 1269 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: No bootstrap required, opened a new log
I20260808 18:33:53.776687 1268 tablet_bootstrap.cc:492] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: No bootstrap required, opened a new log
I20260808 18:33:53.776849 1270 ts_tablet_manager.cc:1397] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:53.776862 1268 ts_tablet_manager.cc:1397] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:53.776862 1269 ts_tablet_manager.cc:1397] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:33:53.780112 1270 raft_consensus.cc:348] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.780375 1270 raft_consensus.cc:374] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:53.780433 1270 raft_consensus.cc:729] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254e279ee4c4494b833f8582f29df43b, State: Initialized, Role: FOLLOWER
I20260808 18:33:53.780939 1270 consensus_queue.cc:260] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.780854 1268 raft_consensus.cc:348] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.780800 1269 raft_consensus.cc:348] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.781091 1269 raft_consensus.cc:374] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:53.781090 1268 raft_consensus.cc:374] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:53.781157 1268 raft_consensus.cc:729] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c786a553e8964e3791991bd2d3dc0316, State: Initialized, Role: FOLLOWER
I20260808 18:33:53.781164 1269 raft_consensus.cc:729] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a486a1773545b5bfe6d3e9e0ff2c88, State: Initialized, Role: FOLLOWER
I20260808 18:33:53.781618 1269 consensus_queue.cc:260] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.781605 1268 consensus_queue.cc:260] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.781994 1270 ts_tablet_manager.cc:1428] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:33:53.782399 1106 heartbeater.cc:503] Master 127.27.251.254:44071 was elected leader, sending a full tablet report...
I20260808 18:33:53.782600 1269 ts_tablet_manager.cc:1428] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:53.782600 1268 ts_tablet_manager.cc:1428] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:33:53.782934 959 heartbeater.cc:503] Master 127.27.251.254:44071 was elected leader, sending a full tablet report...
I20260808 18:33:53.783530 1248 heartbeater.cc:503] Master 127.27.251.254:44071 was elected leader, sending a full tablet report...
I20260808 18:33:53.819965 1276 raft_consensus.cc:482] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:53.820237 1276 raft_consensus.cc:504] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.821519 1276 leader_election.cc:288] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 254e279ee4c4494b833f8582f29df43b (127.27.251.194:41365), e7a486a1773545b5bfe6d3e9e0ff2c88 (127.27.251.195:39111)
I20260808 18:33:53.825976 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.821870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:44793 (local address 127.27.251.194:41365)
0808 18:33:53.822104 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.822109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.822212 (+ 103us) server_negotiation.cc:407] Connection header received
0808 18:33:53.822388 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.822393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.822446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.822538 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:53.822859 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.823437 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.824848 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.825395 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.825528 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.825590 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.825758 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.825842 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260808 18:33:53.826037 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.822011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:41217 (local address 127.27.251.195:39111)
0808 18:33:53.822163 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.822169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.822410 (+ 241us) server_negotiation.cc:407] Connection header received
0808 18:33:53.822557 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.822560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.822612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.822711 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:33:53.823044 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.823571 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.824928 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.825403 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.825536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.825589 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.825829 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.825901 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:53.825994 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.821745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:41365 (local address 127.27.251.193:44793)
0808 18:33:53.822095 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.822111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.822264 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.822556 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.822559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.822579 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:53.822737 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.822744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.823604 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.823610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.824747 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.824758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.825528 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.825532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.825630 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.825656 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.825719 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.825807 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":59,"thread_start_us":128,"threads_started":1}
I20260808 18:33:53.826064 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.821888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39111 (local address 127.27.251.193:41217)
0808 18:33:53.822283 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.822294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:53.822422 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:53.822732 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:53.822734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:53.822749 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:33:53.822877 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.822885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.823705 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.823709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.824773 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:53.824784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.825547 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.825551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:53.825678 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.825696 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:33:53.825794 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 18:33:53.825857 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260808 18:33:53.826751 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da8f75d7a384465b8e95d5fab0ecb51" candidate_uuid: "c786a553e8964e3791991bd2d3dc0316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254e279ee4c4494b833f8582f29df43b" is_pre_election: true
I20260808 18:33:53.827109 1056 raft_consensus.cc:2454] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c786a553e8964e3791991bd2d3dc0316 in term 0.
I20260808 18:33:53.827183 1201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da8f75d7a384465b8e95d5fab0ecb51" candidate_uuid: "c786a553e8964e3791991bd2d3dc0316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" is_pre_election: true
I20260808 18:33:53.827471 1201 raft_consensus.cc:2454] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c786a553e8964e3791991bd2d3dc0316 in term 0.
I20260808 18:33:53.827764 831 leader_election.cc:302] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254e279ee4c4494b833f8582f29df43b, c786a553e8964e3791991bd2d3dc0316; no voters:
I20260808 18:33:53.828079 1276 raft_consensus.cc:2781] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:53.828173 1276 raft_consensus.cc:482] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:53.828228 1276 raft_consensus.cc:3037] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:53.829524 1276 raft_consensus.cc:504] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.830222 1201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da8f75d7a384465b8e95d5fab0ecb51" candidate_uuid: "c786a553e8964e3791991bd2d3dc0316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88"
I20260808 18:33:53.830248 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da8f75d7a384465b8e95d5fab0ecb51" candidate_uuid: "c786a553e8964e3791991bd2d3dc0316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254e279ee4c4494b833f8582f29df43b"
I20260808 18:33:53.830353 1201 raft_consensus.cc:3037] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:53.830381 1056 raft_consensus.cc:3037] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:53.831538 1201 raft_consensus.cc:2454] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c786a553e8964e3791991bd2d3dc0316 in term 1.
I20260808 18:33:53.831733 1276 leader_election.cc:288] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [CANDIDATE]: Term 1 election: Requested vote from peers 254e279ee4c4494b833f8582f29df43b (127.27.251.194:41365), e7a486a1773545b5bfe6d3e9e0ff2c88 (127.27.251.195:39111)
I20260808 18:33:53.831794 1056 raft_consensus.cc:2454] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c786a553e8964e3791991bd2d3dc0316 in term 1.
I20260808 18:33:53.831871 832 leader_election.cc:302] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c786a553e8964e3791991bd2d3dc0316, e7a486a1773545b5bfe6d3e9e0ff2c88; no voters:
I20260808 18:33:53.832132 1276 raft_consensus.cc:2781] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:53.832386 1276 raft_consensus.cc:686] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 1 LEADER]: Becoming Leader. State: Replica: c786a553e8964e3791991bd2d3dc0316, State: Running, Role: LEADER
I20260808 18:33:53.832767 1276 consensus_queue.cc:237] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } }
I20260808 18:33:53.836113 746 catalog_manager.cc:5495] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 reported cstate change: term changed from 0 to 1, leader changed from <none> to c786a553e8964e3791991bd2d3dc0316 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "c786a553e8964e3791991bd2d3dc0316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c786a553e8964e3791991bd2d3dc0316" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 } health_report { overall_health: UNKNOWN } } }
I20260808 18:33:53.858496 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.854998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.27.251.193:32957)
0808 18:33:53.855144 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:53.855148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:53.855178 (+ 30us) server_negotiation.cc:407] Connection header received
0808 18:33:53.855252 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:53.855254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:53.855315 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:53.855399 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:33:53.855691 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.856201 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.857136 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:53.857345 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:53.857423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:53.857736 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:33:53.858177 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:33:53.858271 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:53.858332 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 18:33:53.858382 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:33:53.864569 1056 raft_consensus.cc:1261] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 1 FOLLOWER]: Refusing update from remote peer c786a553e8964e3791991bd2d3dc0316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:53.864558 1201 raft_consensus.cc:1261] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 1 FOLLOWER]: Refusing update from remote peer c786a553e8964e3791991bd2d3dc0316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:53.865309 1279 consensus_queue.cc:1035] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254e279ee4c4494b833f8582f29df43b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:53.865473 1276 consensus_queue.cc:1035] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a486a1773545b5bfe6d3e9e0ff2c88" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:53.873502 1284 mvcc.cc:205] Tried to move back new op lower bound from 7316332682701795328 to 7316332682582204416. Current Snapshot: MvccSnapshot[applied={T|T < 7316332682701795328}]
I20260808 18:33:53.874784 1285 mvcc.cc:205] Tried to move back new op lower bound from 7316332682701795328 to 7316332682582204416. Current Snapshot: MvccSnapshot[applied={T|T < 7316332682701795328}]
I20260808 18:33:53.881219 746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:53.881399 746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:53.883898 746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a120d115e71c47b89a4cb09645385220: Sending DeleteTablet for 3 replicas of tablet 3da8f75d7a384465b8e95d5fab0ecb51
I20260808 18:33:53.884649 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:53 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:34974
I20260808 18:33:53.885015 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:53 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:36382
I20260808 18:33:53.885187 1293 tablet_replica.cc:330] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: stopping tablet replica
I20260808 18:33:53.885520 1295 tablet_replica.cc:330] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316: stopping tablet replica
I20260808 18:33:53.885546 1293 raft_consensus.cc:2229] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:53.885541 1180 tablet_service.cc:1514] Processing DeleteTablet for tablet 3da8f75d7a384465b8e95d5fab0ecb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:53 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:43438
I20260808 18:33:53.885866 1293 raft_consensus.cc:2258] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:53.885985 1295 raft_consensus.cc:2229] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:53.886371 1295 raft_consensus.cc:2258] T 3da8f75d7a384465b8e95d5fab0ecb51 P c786a553e8964e3791991bd2d3dc0316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:53.886488 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 815
I20260808 18:33:53.887609 1293 ts_tablet_manager.cc:1905] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:53.890166 1296 tablet_replica.cc:330] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: stopping tablet replica
I20260808 18:33:53.890498 1296 raft_consensus.cc:2229] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:53.890777 1293 ts_tablet_manager.cc:1918] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:53.890887 1293 log.cc:1196] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/3da8f75d7a384465b8e95d5fab0ecb51
I20260808 18:33:53.890879 1296 raft_consensus.cc:2258] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:53.891405 1293 ts_tablet_manager.cc:1939] T 3da8f75d7a384465b8e95d5fab0ecb51 P 254e279ee4c4494b833f8582f29df43b: Deleting consensus metadata
I20260808 18:33:53.892685 1296 ts_tablet_manager.cc:1905] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:53.892783 729 catalog_manager.cc:4841] TS 254e279ee4c4494b833f8582f29df43b (127.27.251.194:41365): tablet 3da8f75d7a384465b8e95d5fab0ecb51 (table test-table [id=cc0e728b88bc4786ac681e0751fe0fa4]) successfully deleted
I20260808 18:33:53.894770 1296 ts_tablet_manager.cc:1918] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:53.894850 1296 log.cc:1196] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/3da8f75d7a384465b8e95d5fab0ecb51
I20260808 18:33:53.895181 1296 ts_tablet_manager.cc:1939] T 3da8f75d7a384465b8e95d5fab0ecb51 P e7a486a1773545b5bfe6d3e9e0ff2c88: Deleting consensus metadata
I20260808 18:33:53.896292 730 catalog_manager.cc:4841] TS e7a486a1773545b5bfe6d3e9e0ff2c88 (127.27.251.195:39111): tablet 3da8f75d7a384465b8e95d5fab0ecb51 (table test-table [id=cc0e728b88bc4786ac681e0751fe0fa4]) successfully deleted
W20260808 18:33:53.897220 730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:32957 (error 108)
I20260808 18:33:53.897612 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 966
I20260808 18:33:53.898973 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:53.898677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:32957 (local address 127.0.0.1:36392)
0808 18:33:53.898821 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:53.898896 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 18:33:53.899245 730 catalog_manager.cc:4584] TS c786a553e8964e3791991bd2d3dc0316 (127.27.251.193:32957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3da8f75d7a384465b8e95d5fab0ecb51: Network error: Client connection negotiation failed: client connection to 127.27.251.193:32957: connect: Connection refused (error 111)
I20260808 18:33:53.906344 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1111
I20260808 18:33:53.914448 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 714
2026-08-08T18:33:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3864 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T18:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:53Z Disabled control of system clock
I20260808 18:33:53.948001 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38741
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:33:54.062561 1302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:54.062814 1302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:54.062849 1302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:54.066941 1302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:54.066993 1302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:54.067009 1302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:54.067025 1302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:54.067040 1302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:54.072254 1302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:54.073269 1302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:54.074716 1302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:54.083406 1302 server_base.cc:1034] running on GCE node
W20260808 18:33:54.083379 1309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.083468 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.083636 1311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:54.084313 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:54.085479 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:54.086764 1302 hybrid_clock.cc:648] HybridClock initialized: now 1786214034086748 us; error 28 us; skew 500 ppm
I20260808 18:33:54.089416 1302 webserver.cc:460] Webserver started at http://127.27.251.254:38115/ using document root <none> and password file <none>
I20260808 18:33:54.090233 1302 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:54.090304 1302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:54.090571 1302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:54.092792 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "85d13589def348ef9a9a9fadda78ce41"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "97c6b397382754fdc5702234e3739923"
server_key_iv: "92334824a781a52f3b93550e2d3e0318"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.093401 1302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "85d13589def348ef9a9a9fadda78ce41"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "97c6b397382754fdc5702234e3739923"
server_key_iv: "92334824a781a52f3b93550e2d3e0318"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.097196 1302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:54.099642 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.100976 1302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:54.101133 1302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "85d13589def348ef9a9a9fadda78ce41"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "97c6b397382754fdc5702234e3739923"
server_key_iv: "92334824a781a52f3b93550e2d3e0318"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.101253 1302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:54.105954 1302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:54.106557 1302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:54.106741 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:54.113893 1302 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:38741
I20260808 18:33:54.113934 1372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:38741 every 8 connection(s)
I20260808 18:33:54.114872 1302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:54.118243 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:54.123868 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: Bootstrap starting.
I20260808 18:33:54.124454 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1302
I20260808 18:33:54.124601 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:54.124922 28655 external_mini_cluster.cc:1428] Setting key bdec99bd120d7ed7ef5a081ec959b309
I20260808 18:33:54.126816 1373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:54.127575 1373 log.cc:824] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:54.129904 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: No bootstrap required, opened a new log
I20260808 18:33:54.130607 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.126314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.27.251.254:38741)
0808 18:33:54.127000 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.127018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.127063 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:33:54.127753 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.127772 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.128117 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.128390 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.129237 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.129242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.129253 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.129263 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.129315 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.129330 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.129419 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.129597 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.129886 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":162,"threads_started":1}
I20260808 18:33:54.133771 1373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } }
I20260808 18:33:54.133981 1373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:54.134068 1373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d13589def348ef9a9a9fadda78ce41, State: Initialized, Role: FOLLOWER
I20260808 18:33:54.134563 1373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } }
I20260808 18:33:54.134713 1373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:54.134781 1373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:54.134886 1373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:54.135783 1373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } }
I20260808 18:33:54.136132 1373 leader_election.cc:302] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85d13589def348ef9a9a9fadda78ce41; no voters:
I20260808 18:33:54.136449 1373 leader_election.cc:288] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:54.136560 1379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:54.136857 1379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [term 1 LEADER]: Becoming Leader. State: Replica: 85d13589def348ef9a9a9fadda78ce41, State: Running, Role: LEADER
I20260808 18:33:54.137183 1379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } }
I20260808 18:33:54.137749 1373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:54.138753 1380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85d13589def348ef9a9a9fadda78ce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } } }
I20260808 18:33:54.138900 1380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:54.139245 1388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:54.140766 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85d13589def348ef9a9a9fadda78ce41. Latest consensus state: current_term: 1 leader_uuid: "85d13589def348ef9a9a9fadda78ce41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85d13589def348ef9a9a9fadda78ce41" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38741 } } }
I20260808 18:33:54.140869 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:54.142422 1388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:54.148666 1388 catalog_manager.cc:1347] Generated new cluster ID: b6cef9a64a2b45b487a6fb2c867e2306
I20260808 18:33:54.148730 1388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:54.160880 1388 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:54.161247 1388 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:54.172219 1388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: Generated new TSK 0
I20260808 18:33:54.173108 1388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:54.185978 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--builtin_ntp_servers=127.27.251.212:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:33:54.322297 1399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:54.322597 1399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:54.322678 1399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:54.326895 1399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:54.326977 1399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:54.327076 1399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:54.331920 1399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:54.333227 1399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:54.334671 1399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:54.344870 1405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.344914 1404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.345275 1407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:54.345273 1399 server_base.cc:1034] running on GCE node
I20260808 18:33:54.345877 1399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:54.346824 1399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:54.348065 1399 hybrid_clock.cc:648] HybridClock initialized: now 1786214034348024 us; error 50 us; skew 500 ppm
I20260808 18:33:54.350768 1399 webserver.cc:460] Webserver started at http://127.27.251.193:46267/ using document root <none> and password file <none>
I20260808 18:33:54.351675 1399 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:54.351788 1399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:54.352090 1399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:54.354547 1399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "efeb6a3adaa34edab356252972a8300b"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "a83f1bf53ad8b83518d233bc7937433b"
server_key_iv: "c5864a0e1448c12b926faebb847ec4f5"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.355139 1399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "efeb6a3adaa34edab356252972a8300b"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "a83f1bf53ad8b83518d233bc7937433b"
server_key_iv: "c5864a0e1448c12b926faebb847ec4f5"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.359527 1399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:33:54.362190 1413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.363458 1399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:54.363564 1399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "efeb6a3adaa34edab356252972a8300b"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "a83f1bf53ad8b83518d233bc7937433b"
server_key_iv: "c5864a0e1448c12b926faebb847ec4f5"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.363660 1399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:54.368956 1399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:54.369540 1399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:54.369686 1399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:54.370236 1399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:54.371177 1399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:54.371225 1399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.371273 1399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:54.371289 1399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.380316 1399 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:39133
I20260808 18:33:54.380386 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:39133 every 8 connection(s)
I20260808 18:33:54.381343 1399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:54.383906 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1399
I20260808 18:33:54.384047 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:54.384413 28655 external_mini_cluster.cc:1428] Setting key 821531df10f2921f32f81996531d6911
I20260808 18:33:54.386948 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--builtin_ntp_servers=127.27.251.212:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 18:33:54.388777 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.383349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:55077 (local address 127.27.251.254:38741)
0808 18:33:54.383520 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.383525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.384862 (+ 1337us) server_negotiation.cc:407] Connection header received
0808 18:33:54.386824 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.386827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.386864 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.386943 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.388286 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.388295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.388297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.388309 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.388329 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.388336 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.388413 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.388630 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.388672 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:33:54.390170 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.383757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38741 (local address 127.27.251.193:55077)
0808 18:33:54.384683 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.384748 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.386563 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.387164 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.387177 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.387300 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.387928 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.387932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.388005 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.388109 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.388439 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.388451 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.388460 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.388686 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.388991 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":149,"threads_started":1}
I20260808 18:33:54.391543 1534 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38741
I20260808 18:33:54.391791 1534 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:54.392231 1534 heartbeater.cc:511] Master 127.27.251.254:38741 requested a full tablet report, sending...
I20260808 18:33:54.393934 1336 ts_manager.cc:194] Registered new tserver with Master: efeb6a3adaa34edab356252972a8300b (127.27.251.193:39133)
W20260808 18:33:54.510510 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:54.510825 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:54.510865 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:54.515251 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:54.515304 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:54.515381 1537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:54.520177 1537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:54.521318 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:54.522749 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:54.532752 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.532771 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.532758 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:54.532953 1537 server_base.cc:1034] running on GCE node
I20260808 18:33:54.533404 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:54.534142 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:54.535313 1537 hybrid_clock.cc:648] HybridClock initialized: now 1786214034535295 us; error 27 us; skew 500 ppm
I20260808 18:33:54.537307 1537 webserver.cc:460] Webserver started at http://127.27.251.194:37837/ using document root <none> and password file <none>
I20260808 18:33:54.537950 1537 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:54.538017 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:54.538275 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:54.540416 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "5fa63b943ee748ccbca34bc48e8aece9"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "bc061e71bee41533e937177fad8cbf5e"
server_key_iv: "f27d5eb3bc916283ee4ee59889f8865c"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.541074 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa63b943ee748ccbca34bc48e8aece9"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "bc061e71bee41533e937177fad8cbf5e"
server_key_iv: "f27d5eb3bc916283ee4ee59889f8865c"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.544937 1537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:54.547554 1552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.548947 1537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:54.549081 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "5fa63b943ee748ccbca34bc48e8aece9"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "bc061e71bee41533e937177fad8cbf5e"
server_key_iv: "f27d5eb3bc916283ee4ee59889f8865c"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.549194 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:54.553964 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:54.554589 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:54.554773 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:54.555326 1537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:54.556268 1537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:54.556315 1537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.556383 1537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:54.556427 1537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.565349 1537 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:39637
I20260808 18:33:54.565378 1674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:39637 every 8 connection(s)
I20260808 18:33:54.566252 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:54.572613 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.568286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:45167 (local address 127.27.251.254:38741)
0808 18:33:54.568458 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.568463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.569664 (+ 1201us) server_negotiation.cc:407] Connection header received
0808 18:33:54.570941 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.570944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.570983 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.571062 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.572214 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.572217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.572219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.572228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.572243 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.572250 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.572322 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.572426 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.572462 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:54.573498 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.568830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38741 (local address 127.27.251.194:45167)
0808 18:33:54.569522 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.569563 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.570751 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.571230 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.571236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.571322 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.571967 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.571969 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.572014 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.572083 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.572319 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.572323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.572327 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.572446 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.572742 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260808 18:33:54.574774 1675 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38741
I20260808 18:33:54.575009 1675 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:54.575423 1675 heartbeater.cc:511] Master 127.27.251.254:38741 requested a full tablet report, sending...
I20260808 18:33:54.575996 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1537
I20260808 18:33:54.576117 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:54.576476 28655 external_mini_cluster.cc:1428] Setting key 962c345b94ce3f19c31d3d5587a69574
I20260808 18:33:54.576655 1336 ts_manager.cc:194] Registered new tserver with Master: 5fa63b943ee748ccbca34bc48e8aece9 (127.27.251.194:39637)
I20260808 18:33:54.578457 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--builtin_ntp_servers=127.27.251.212:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 18:33:54.693670 1680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:54.693943 1680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:54.693984 1680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:54.698189 1680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:54.698244 1680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:54.698302 1680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:54.702961 1680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:54.704020 1680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:54.705675 1680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:54.715547 1686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.715651 1687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:54.715708 1689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:54.715945 1680 server_base.cc:1034] running on GCE node
I20260808 18:33:54.716405 1680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:54.717069 1680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:54.718261 1680 hybrid_clock.cc:648] HybridClock initialized: now 1786214034718249 us; error 36 us; skew 500 ppm
I20260808 18:33:54.720153 1680 webserver.cc:460] Webserver started at http://127.27.251.195:36923/ using document root <none> and password file <none>
I20260808 18:33:54.720810 1680 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:54.720904 1680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:54.721141 1680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:54.723269 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "ec43ecdb2e08433aac6dd6a063330b1f"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "0c9c9e2a2c07d60f240471eba6ff6329"
server_key_iv: "54a9fa290b2c7687e51b550a57254842"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.723870 1680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "ec43ecdb2e08433aac6dd6a063330b1f"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "0c9c9e2a2c07d60f240471eba6ff6329"
server_key_iv: "54a9fa290b2c7687e51b550a57254842"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.727756 1680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260808 18:33:54.730406 1695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.731699 1680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:33:54.731796 1680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "ec43ecdb2e08433aac6dd6a063330b1f"
format_stamp: "Formatted at 2026-08-08 18:33:54 on dist-test-slave-fd7l"
server_key: "0c9c9e2a2c07d60f240471eba6ff6329"
server_key_iv: "54a9fa290b2c7687e51b550a57254842"
server_key_version: "encryptionkey@0"
I20260808 18:33:54.731885 1680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:54.736397 1680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:54.737028 1680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:54.737169 1680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:54.737699 1680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:54.738679 1680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:54.738726 1680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.738768 1680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:54.738785 1680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:54.747642 1680 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:34313
I20260808 18:33:54.747666 1815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:34313 every 8 connection(s)
I20260808 18:33:54.748608 1680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:54.754710 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.750462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:56005 (local address 127.27.251.254:38741)
0808 18:33:54.750604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.750608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.751735 (+ 1127us) server_negotiation.cc:407] Connection header received
0808 18:33:54.753088 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.753090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.753178 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.753254 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.754317 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.754320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.754322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.754333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.754351 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.754359 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.754439 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.754563 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.754601 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:33:54.755398 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1680
I20260808 18:33:54.755489 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:54.755707 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.750875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38741 (local address 127.27.251.195:56005)
0808 18:33:54.751579 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.751621 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.752888 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.753436 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.753442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.753530 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.754052 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.754054 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.754094 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.754174 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.754434 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.754441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.754445 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.754575 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.754794 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260808 18:33:54.755810 28655 external_mini_cluster.cc:1428] Setting key 26b6b400062dfc250e2e5bc18cd54903
I20260808 18:33:54.756924 1816 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38741
I20260808 18:33:54.757203 1816 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:54.757668 1816 heartbeater.cc:511] Master 127.27.251.254:38741 requested a full tablet report, sending...
I20260808 18:33:54.758687 1335 ts_manager.cc:194] Registered new tserver with Master: ec43ecdb2e08433aac6dd6a063330b1f (127.27.251.195:34313)
I20260808 18:33:54.769663 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:33:54.773473 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.771644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.27.251.254:38741)
0808 18:33:54.771776 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.771780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.771999 (+ 219us) server_negotiation.cc:407] Connection header received
0808 18:33:54.772108 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.772110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.772145 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.772213 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.773064 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.773067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.773069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.773080 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.773098 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.773106 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.773192 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.773332 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.773372 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:54.776500 1335 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:33:54.779616 1335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:33:54.791347 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.788169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.27.251.195:34313)
0808 18:33:54.788922 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.788935 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.788961 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:33:54.789018 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.789024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.789242 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.789400 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.790685 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.790691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.790712 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.790722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.790758 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.790781 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.790903 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.791108 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.791184 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":130,"threads_started":1}
I20260808 18:33:54.791334 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.788039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:34313 (local address 127.0.0.1:37332)
0808 18:33:54.788565 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.788613 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.788715 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.789750 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.789760 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.789840 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.790425 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.790428 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.790451 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.790533 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.790952 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.790965 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.790974 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.791092 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.791170 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":111,"threads_started":1}
I20260808 18:33:54.791524 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.788121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:39133 (local address 127.0.0.1:35074)
0808 18:33:54.789471 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.789484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.789685 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.790161 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.790164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.790176 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.790530 (+ 354us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.790532 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.790540 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.790557 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.791058 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.791064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.791068 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.791148 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.791191 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":94,"threads_started":1}
I20260808 18:33:54.792198 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.788894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.27.251.193:39133)
0808 18:33:54.789345 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.789367 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.789616 (+ 249us) server_negotiation.cc:407] Connection header received
0808 18:33:54.789828 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.789835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.790014 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.790166 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.790795 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.790804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.790817 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.790828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.790866 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.790892 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.790996 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.791868 (+ 872us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.791940 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260808 18:33:54.792932 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.789159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:39637 (local address 127.0.0.1:56398)
0808 18:33:54.789516 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.789530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.789600 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.790861 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.790864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.790878 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.791540 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.791542 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.791552 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.791579 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.792696 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.792699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.792702 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.792784 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.792819 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":98,"threads_started":1}
I20260808 18:33:54.793056 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.789471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.27.251.194:39637)
0808 18:33:54.789955 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.789966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.789988 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:33:54.790038 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.790043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.790714 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.790858 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.792428 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.792433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.792443 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.792454 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.792489 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.792543 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.792646 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.792829 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.792901 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":128,"threads_started":1}
I20260808 18:33:54.794946 1465 tablet_service.cc:1467] Processing CreateTablet for tablet 7cba44b8eacc49deb2017f283c6b108d (DEFAULT_TABLE table=test-table [id=98512e6e79f04dd9b83b1bba523794cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:54.794947 1745 tablet_service.cc:1467] Processing CreateTablet for tablet 7cba44b8eacc49deb2017f283c6b108d (DEFAULT_TABLE table=test-table [id=98512e6e79f04dd9b83b1bba523794cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:54.796113 1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cba44b8eacc49deb2017f283c6b108d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:54.796114 1745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cba44b8eacc49deb2017f283c6b108d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:54.801178 1602 tablet_service.cc:1467] Processing CreateTablet for tablet 7cba44b8eacc49deb2017f283c6b108d (DEFAULT_TABLE table=test-table [id=98512e6e79f04dd9b83b1bba523794cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:33:54.802134 1602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cba44b8eacc49deb2017f283c6b108d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:54.804607 1835 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Bootstrap starting.
I20260808 18:33:54.805349 1834 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Bootstrap starting.
I20260808 18:33:54.807195 1835 tablet_bootstrap.cc:654] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:54.807302 1834 tablet_bootstrap.cc:654] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:54.808060 1834 log.cc:824] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:54.808128 1835 log.cc:824] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:54.810073 1838 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Bootstrap starting.
I20260808 18:33:54.810344 1834 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: No bootstrap required, opened a new log
I20260808 18:33:54.810566 1834 ts_tablet_manager.cc:1397] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:33:54.812115 1838 tablet_bootstrap.cc:654] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:54.812930 1838 log.cc:824] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:54.814018 1835 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: No bootstrap required, opened a new log
I20260808 18:33:54.814235 1835 ts_tablet_manager.cc:1397] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 18:33:54.814697 1838 tablet_bootstrap.cc:492] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: No bootstrap required, opened a new log
I20260808 18:33:54.814880 1838 ts_tablet_manager.cc:1397] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:33:54.814740 1834 raft_consensus.cc:348] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.815023 1834 raft_consensus.cc:374] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:54.815130 1834 raft_consensus.cc:729] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeb6a3adaa34edab356252972a8300b, State: Initialized, Role: FOLLOWER
I20260808 18:33:54.815640 1834 consensus_queue.cc:260] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.816500 1534 heartbeater.cc:503] Master 127.27.251.254:38741 was elected leader, sending a full tablet report...
I20260808 18:33:54.816715 1834 ts_tablet_manager.cc:1428] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:33:54.818885 1835 raft_consensus.cc:348] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.819183 1835 raft_consensus.cc:374] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:54.819303 1835 raft_consensus.cc:729] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec43ecdb2e08433aac6dd6a063330b1f, State: Initialized, Role: FOLLOWER
I20260808 18:33:54.819849 1835 consensus_queue.cc:260] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.820501 1816 heartbeater.cc:503] Master 127.27.251.254:38741 was elected leader, sending a full tablet report...
I20260808 18:33:54.820946 1838 raft_consensus.cc:348] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.821205 1838 raft_consensus.cc:374] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:54.821264 1838 raft_consensus.cc:729] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa63b943ee748ccbca34bc48e8aece9, State: Initialized, Role: FOLLOWER
I20260808 18:33:54.821707 1838 consensus_queue.cc:260] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.821951 1835 ts_tablet_manager.cc:1428] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260808 18:33:54.822444 1675 heartbeater.cc:503] Master 127.27.251.254:38741 was elected leader, sending a full tablet report...
I20260808 18:33:54.822647 1838 ts_tablet_manager.cc:1428] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260808 18:33:54.882588 1535 tablet.cc:2369] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:54.982405 1840 raft_consensus.cc:482] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:33:54.982672 1840 raft_consensus.cc:504] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.983904 1840 leader_election.cc:288] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec43ecdb2e08433aac6dd6a063330b1f (127.27.251.195:34313), 5fa63b943ee748ccbca34bc48e8aece9 (127.27.251.194:39637)
I20260808 18:33:54.986254 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.984342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:34027 (local address 127.27.251.195:34313)
0808 18:33:54.984534 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.984540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.984699 (+ 159us) server_negotiation.cc:407] Connection header received
0808 18:33:54.984860 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.984864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.984907 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.984993 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.985850 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.985854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.985856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.985868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.985895 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.985903 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.985996 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.986082 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.986123 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 18:33:54.986274 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.984185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:34313 (local address 127.27.251.193:34027)
0808 18:33:54.984597 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.984610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.984741 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.985027 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.985035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.985049 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.985681 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.985683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.985693 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.985717 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.985981 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.985984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.985987 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.986074 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.986117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":138,"threads_started":1}
I20260808 18:33:54.986420 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.984835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:35927 (local address 127.27.251.194:39637)
0808 18:33:54.984981 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:54.984988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:54.985135 (+ 147us) server_negotiation.cc:407] Connection header received
0808 18:33:54.985193 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:54.985197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:54.985244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:54.985333 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:54.986037 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:54.986040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:54.986043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:54.986055 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:54.986082 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:54.986090 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:54.986189 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:54.986253 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 18:33:54.986294 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:33:54.986457 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:54.984690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:39637 (local address 127.27.251.193:35927)
0808 18:33:54.985014 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:54.985029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:54.985124 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:54.985377 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:54.985381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:54.985397 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:54.985849 (+ 452us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:33:54.985851 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:54.985862 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:33:54.985896 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:54.986171 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:54.986174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:54.986178 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:33:54.986260 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:33:54.986293 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":66,"threads_started":1}
I20260808 18:33:54.987110 1768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cba44b8eacc49deb2017f283c6b108d" candidate_uuid: "efeb6a3adaa34edab356252972a8300b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" is_pre_election: true
I20260808 18:33:54.987110 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cba44b8eacc49deb2017f283c6b108d" candidate_uuid: "efeb6a3adaa34edab356252972a8300b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa63b943ee748ccbca34bc48e8aece9" is_pre_election: true
I20260808 18:33:54.987416 1768 raft_consensus.cc:2454] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb6a3adaa34edab356252972a8300b in term 0.
I20260808 18:33:54.987418 1624 raft_consensus.cc:2454] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efeb6a3adaa34edab356252972a8300b in term 0.
I20260808 18:33:54.987922 1418 leader_election.cc:302] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa63b943ee748ccbca34bc48e8aece9, efeb6a3adaa34edab356252972a8300b; no voters:
I20260808 18:33:54.988212 1840 raft_consensus.cc:2781] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:33:54.988306 1840 raft_consensus.cc:482] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:33:54.988353 1840 raft_consensus.cc:3037] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:54.989522 1840 raft_consensus.cc:504] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.989858 1840 leader_election.cc:288] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [CANDIDATE]: Term 1 election: Requested vote from peers ec43ecdb2e08433aac6dd6a063330b1f (127.27.251.195:34313), 5fa63b943ee748ccbca34bc48e8aece9 (127.27.251.194:39637)
I20260808 18:33:54.990269 1768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cba44b8eacc49deb2017f283c6b108d" candidate_uuid: "efeb6a3adaa34edab356252972a8300b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec43ecdb2e08433aac6dd6a063330b1f"
I20260808 18:33:54.990286 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cba44b8eacc49deb2017f283c6b108d" candidate_uuid: "efeb6a3adaa34edab356252972a8300b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa63b943ee748ccbca34bc48e8aece9"
I20260808 18:33:54.990377 1768 raft_consensus.cc:3037] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:54.990425 1624 raft_consensus.cc:3037] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:54.991521 1768 raft_consensus.cc:2454] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb6a3adaa34edab356252972a8300b in term 1.
I20260808 18:33:54.991708 1624 raft_consensus.cc:2454] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeb6a3adaa34edab356252972a8300b in term 1.
I20260808 18:33:54.991819 1418 leader_election.cc:302] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec43ecdb2e08433aac6dd6a063330b1f, efeb6a3adaa34edab356252972a8300b; no voters:
I20260808 18:33:54.992103 1840 raft_consensus.cc:2781] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:54.992278 1840 raft_consensus.cc:686] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 1 LEADER]: Becoming Leader. State: Replica: efeb6a3adaa34edab356252972a8300b, State: Running, Role: LEADER
I20260808 18:33:54.992666 1840 consensus_queue.cc:237] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } }
I20260808 18:33:54.995994 1336 catalog_manager.cc:5495] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b reported cstate change: term changed from 0 to 1, leader changed from <none> to efeb6a3adaa34edab356252972a8300b (127.27.251.193). New cstate: current_term: 1 leader_uuid: "efeb6a3adaa34edab356252972a8300b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efeb6a3adaa34edab356252972a8300b" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 39133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 } health_report { overall_health: UNKNOWN } } }
W20260808 18:33:54.999586 1817 tablet.cc:2369] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:33:55.026978 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.025575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.27.251.193:39133)
0808 18:33:55.025714 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:55.025718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:55.025733 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:33:55.025811 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:55.025813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:55.025871 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:55.025948 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:55.026656 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:55.026658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:55.026660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:55.026668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:55.026683 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:55.026689 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:55.026756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:55.026836 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 18:33:55.026871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:33:55.032914 1624 raft_consensus.cc:1261] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 1 FOLLOWER]: Refusing update from remote peer efeb6a3adaa34edab356252972a8300b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:55.032922 1768 raft_consensus.cc:1261] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 1 FOLLOWER]: Refusing update from remote peer efeb6a3adaa34edab356252972a8300b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:33:55.033664 1845 consensus_queue.cc:1035] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa63b943ee748ccbca34bc48e8aece9" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:55.033833 1840 consensus_queue.cc:1035] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec43ecdb2e08433aac6dd6a063330b1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:33:55.040648 1848 mvcc.cc:205] Tried to move back new op lower bound from 7316332687487225856 to 7316332687332823040. Current Snapshot: MvccSnapshot[applied={T|T < 7316332687487225856}]
I20260808 18:33:55.042456 1850 mvcc.cc:205] Tried to move back new op lower bound from 7316332687487225856 to 7316332687332823040. Current Snapshot: MvccSnapshot[applied={T|T < 7316332687487225856}]
I20260808 18:33:55.049022 1335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:55.049201 1335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:33:55.051733 1335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85d13589def348ef9a9a9fadda78ce41: Sending DeleteTablet for 3 replicas of tablet 7cba44b8eacc49deb2017f283c6b108d
I20260808 18:33:55.052554 1465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cba44b8eacc49deb2017f283c6b108d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:55 UTC) from {username='slave'} at 127.0.0.1:35074
I20260808 18:33:55.052556 1602 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cba44b8eacc49deb2017f283c6b108d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:55 UTC) from {username='slave'} at 127.0.0.1:56398
I20260808 18:33:55.052644 1745 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cba44b8eacc49deb2017f283c6b108d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:33:55 UTC) from {username='slave'} at 127.0.0.1:37332
I20260808 18:33:55.053045 1860 tablet_replica.cc:330] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b: stopping tablet replica
I20260808 18:33:55.053063 1859 tablet_replica.cc:330] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: stopping tablet replica
I20260808 18:33:55.053370 1859 raft_consensus.cc:2229] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:55.053471 1860 raft_consensus.cc:2229] T 7cba44b8eacc49deb2017f283c6b108d P efeb6a3adaa34edab356252972a8300b [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:33:55.053718 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1399
I20260808 18:33:55.053726 1859 raft_consensus.cc:2258] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:55.055724 1859 ts_tablet_manager.cc:1905] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:55.055876 1861 tablet_replica.cc:330] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: stopping tablet replica
I20260808 18:33:55.056164 1861 raft_consensus.cc:2229] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:33:55.056447 1861 raft_consensus.cc:2258] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:33:55.058334 1859 ts_tablet_manager.cc:1918] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:55.058457 1859 log.cc:1196] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/7cba44b8eacc49deb2017f283c6b108d
I20260808 18:33:55.058883 1859 ts_tablet_manager.cc:1939] T 7cba44b8eacc49deb2017f283c6b108d P 5fa63b943ee748ccbca34bc48e8aece9: Deleting consensus metadata
I20260808 18:33:55.059685 1861 ts_tablet_manager.cc:1905] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:33:55.060025 1322 catalog_manager.cc:4841] TS 5fa63b943ee748ccbca34bc48e8aece9 (127.27.251.194:39637): tablet 7cba44b8eacc49deb2017f283c6b108d (table test-table [id=98512e6e79f04dd9b83b1bba523794cc]) successfully deleted
I20260808 18:33:55.063167 1861 ts_tablet_manager.cc:1918] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:33:55.063277 1861 log.cc:1196] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/7cba44b8eacc49deb2017f283c6b108d
I20260808 18:33:55.063629 1861 ts_tablet_manager.cc:1939] T 7cba44b8eacc49deb2017f283c6b108d P ec43ecdb2e08433aac6dd6a063330b1f: Deleting consensus metadata
W20260808 18:33:55.063917 1319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:39133 (error 108)
I20260808 18:33:55.064829 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1537
I20260808 18:33:55.065102 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.064757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:39133 (local address 127.0.0.1:35090)
0808 18:33:55.064914 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:55.064993 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 18:33:55.065254 1319 catalog_manager.cc:4584] TS efeb6a3adaa34edab356252972a8300b (127.27.251.193:39133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cba44b8eacc49deb2017f283c6b108d: Network error: Client connection negotiation failed: client connection to 127.27.251.193:39133: connect: Connection refused (error 111)
I20260808 18:33:55.069082 1322 catalog_manager.cc:4841] TS ec43ecdb2e08433aac6dd6a063330b1f (127.27.251.195:34313): tablet 7cba44b8eacc49deb2017f283c6b108d (table test-table [id=98512e6e79f04dd9b83b1bba523794cc]) successfully deleted
I20260808 18:33:55.074915 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1680
I20260808 18:33:55.082330 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1302
2026-08-08T18:33:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1167 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 18:33:55.106796 28655 mini_oidc.cc:152] Starting JWKS server
I20260808 18:33:55.108675 28655 webserver.cc:460] Webserver started at https://127.0.0.1:36153/ using document root <none> and password file <none>
I20260808 18:33:55.108870 28655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:33:55.109157 28655 webserver.cc:460] Webserver started at http://127.0.0.1:34963/ using document root <none> and password file <none>
2026-08-08T18:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:55Z Disabled control of system clock
I20260808 18:33:55.119692 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37215
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:37215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36153/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/testchainca.cert with env {}
W20260808 18:33:55.236475 1868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:55.236769 1868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:55.236809 1868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:55.240793 1868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:55.240856 1868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:55.240872 1868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:55.240895 1868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:55.240911 1868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:33:55.240926 1868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:33:55.240940 1868 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36153/jwks.json
I20260808 18:33:55.246820 1868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:37215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36153/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:55.247841 1868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:55.248914 1868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:33:55.257968 1868 server_base.cc:1034] running on GCE node
W20260808 18:33:55.257921 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:55.257956 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:55.257926 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:55.258939 1868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:55.260090 1868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:55.261273 1868 hybrid_clock.cc:648] HybridClock initialized: now 1786214035261257 us; error 29 us; skew 500 ppm
I20260808 18:33:55.263254 1868 webserver.cc:460] Webserver started at http://127.27.251.254:42139/ using document root <none> and password file <none>
I20260808 18:33:55.263856 1868 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:55.263957 1868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:55.264199 1868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:55.266306 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "73bc03110d714dbea4348fb3a065c46b"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "68ea11a3d25fa6ca77561ca2a9cb2492"
server_key_iv: "c1e8aedb97cfe88f1b3254ec1468a201"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.266899 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "73bc03110d714dbea4348fb3a065c46b"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "68ea11a3d25fa6ca77561ca2a9cb2492"
server_key_iv: "c1e8aedb97cfe88f1b3254ec1468a201"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.270745 1868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:55.273197 1883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:55.274501 1868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:55.274659 1868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "73bc03110d714dbea4348fb3a065c46b"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "68ea11a3d25fa6ca77561ca2a9cb2492"
server_key_iv: "c1e8aedb97cfe88f1b3254ec1468a201"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.274777 1868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:55.305516 1868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:55.306285 1868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:55.306507 1868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:55.314000 1868 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:37215
I20260808 18:33:55.314013 1944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:37215 every 8 connection(s)
I20260808 18:33:55.314971 1868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:55.316244 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1868
I20260808 18:33:55.316380 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:55.316800 28655 external_mini_cluster.cc:1428] Setting key 42c03b89f8758ce05d7c368883e10eb8
I20260808 18:33:55.319412 1945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:55.326015 1945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b: Bootstrap starting.
I20260808 18:33:55.326401 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.318273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.27.251.254:37215)
0808 18:33:55.318872 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:55.318886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:55.318923 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:33:55.319679 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:55.319697 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:55.320115 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:55.320395 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:55.320808 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.321770 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.323007 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.323475 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.323594 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.324611 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:55.324617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:55.324629 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:55.325028 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:55.325082 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:55.325103 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:55.325180 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:55.325354 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 18:33:55.325645 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260808 18:33:55.328936 1945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:55.329535 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:55.329859 1945 log.cc:824] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:55.332072 1945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b: No bootstrap required, opened a new log
I20260808 18:33:55.335743 1945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } }
I20260808 18:33:55.335958 1945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:55.335994 1945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73bc03110d714dbea4348fb3a065c46b, State: Initialized, Role: FOLLOWER
I20260808 18:33:55.336445 1945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } }
I20260808 18:33:55.336637 1945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:55.336692 1945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:55.336766 1945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:55.337682 1945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } }
I20260808 18:33:55.338060 1945 leader_election.cc:302] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73bc03110d714dbea4348fb3a065c46b; no voters:
I20260808 18:33:55.338438 1945 leader_election.cc:288] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:55.338549 1950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:55.338809 1950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [term 1 LEADER]: Becoming Leader. State: Replica: 73bc03110d714dbea4348fb3a065c46b, State: Running, Role: LEADER
I20260808 18:33:55.339136 1950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } }
I20260808 18:33:55.339648 1945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:55.340827 1952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73bc03110d714dbea4348fb3a065c46b. Latest consensus state: current_term: 1 leader_uuid: "73bc03110d714dbea4348fb3a065c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } } }
I20260808 18:33:55.340986 1952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:55.341437 1959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:55.342088 1951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73bc03110d714dbea4348fb3a065c46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73bc03110d714dbea4348fb3a065c46b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37215 } } }
I20260808 18:33:55.342239 1951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:55.344125 1959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:55.349408 1959 catalog_manager.cc:1347] Generated new cluster ID: 69a8ae9eb0c54c1a8a97c3c87a398bf0
I20260808 18:33:55.349476 1959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:55.395802 1959 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:55.397152 1959 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:55.404268 1959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73bc03110d714dbea4348fb3a065c46b: Generated new TSK 0
I20260808 18:33:55.405038 1959 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 18:33:55.452705 1975 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 18:33:55.452873 1975 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 18:33:55.453369 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.427832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.27.251.254:37215)
0808 18:33:55.427972 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:55.427979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:55.428221 (+ 242us) server_negotiation.cc:407] Connection header received
0808 18:33:55.428476 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:55.428479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:55.428581 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:55.428701 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:33:55.428974 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.429521 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.430442 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.430665 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.430771 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.431138 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:33:55.452882 (+ 21744us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 18:33:55.453210 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32848: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260808 18:33:55.453451 1948 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32848: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 18:33:55.454756 1974 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:55.427717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:37215 (local address 127.0.0.1:32848)
0808 18:33:55.428133 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:55.428146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:55.428302 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:55.428718 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:55.428721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:55.428729 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:33:55.428859 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:55.428866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.429642 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.429648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:55.430311 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:55.430319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.430790 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.430794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:55.430896 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.430938 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:33:55.453179 (+ 22241us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 18:33:55.453440 (+ 261us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.251.254:37215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.251.254:37215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 18:33:55.455533 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1868
2026-08-08T18:33:55Z chronyd exiting
I20260808 18:33:55.496189 28655 test_util.cc:175] -----------------------------------------------
I20260808 18:33:55.496276 28655 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786214007390461-28655-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (391 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 18:33:55.497466 28655 mini_oidc.cc:152] Starting JWKS server
I20260808 18:33:55.498344 28655 webserver.cc:460] Webserver started at https://127.0.0.1:38949/ using document root <none> and password file <none>
I20260808 18:33:55.498453 28655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:33:55.498740 28655 webserver.cc:460] Webserver started at http://127.0.0.1:43183/ using document root <none> and password file <none>
2026-08-08T18:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:55Z Disabled control of system clock
I20260808 18:33:55.508715 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35063
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38949/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/testchainca.cert with env {}
W20260808 18:33:55.622746 1981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:55.623051 1981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:55.623131 1981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:55.627130 1981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:55.627226 1981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:55.627256 1981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:55.627295 1981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:55.627326 1981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:33:55.627363 1981 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:33:55.627396 1981 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38949/jwks.json
I20260808 18:33:55.632673 1981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38949/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:55.633720 1981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:55.634737 1981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:55.643482 1988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:55.643476 1986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:55.643652 1981 server_base.cc:1034] running on GCE node
W20260808 18:33:55.643472 1990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:55.644333 1981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:55.645465 1981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:55.646657 1981 hybrid_clock.cc:648] HybridClock initialized: now 1786214035646639 us; error 29 us; skew 500 ppm
I20260808 18:33:55.654174 1981 webserver.cc:460] Webserver started at http://127.27.251.254:35981/ using document root <none> and password file <none>
I20260808 18:33:55.654811 1981 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:55.654908 1981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:55.655143 1981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:55.657387 1981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "0fc76273d1978e5c9447300bb15d2408"
server_key_iv: "7371c28b3754c1918ab045abb1e5e35e"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.658049 1981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "0fc76273d1978e5c9447300bb15d2408"
server_key_iv: "7371c28b3754c1918ab045abb1e5e35e"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.661828 1981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:33:55.664368 1996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:55.665699 1981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:55.665848 1981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "0fc76273d1978e5c9447300bb15d2408"
server_key_iv: "7371c28b3754c1918ab045abb1e5e35e"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.665966 1981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:55.685350 1981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:55.686157 1981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:55.686388 1981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:55.694180 1981 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35063
I20260808 18:33:55.694186 2053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35063 every 8 connection(s)
I20260808 18:33:55.695258 1981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:55.699069 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:55.704802 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 1981
I20260808 18:33:55.704948 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:55.705313 28655 external_mini_cluster.cc:1428] Setting key 25ed4859fbbda476be6d1a219b770e22
I20260808 18:33:55.705618 2054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc: Bootstrap starting.
I20260808 18:33:55.708465 2054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:55.709358 2054 log.cc:824] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:55.711458 2054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc: No bootstrap required, opened a new log
I20260808 18:33:55.714942 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.706807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.27.251.254:35063)
0808 18:33:55.707563 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:55.707578 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:55.707615 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:33:55.708311 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:55.708329 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:55.708842 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:55.709221 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:55.709690 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.710654 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.711591 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.712079 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.712254 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.713170 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:55.713175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:55.713187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:55.713617 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:55.713676 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:55.713696 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:55.713806 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:55.713930 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 18:33:55.714202 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":179,"threads_started":1}
I20260808 18:33:55.715608 2054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } }
I20260808 18:33:55.715883 2054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:55.716012 2054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5cc1ee11294e1eaa0dacf27ae352cc, State: Initialized, Role: FOLLOWER
I20260808 18:33:55.716586 2054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } }
I20260808 18:33:55.716768 2054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:55.716847 2054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:55.716969 2054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:55.717916 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:55.718087 2054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } }
I20260808 18:33:55.718482 2054 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5cc1ee11294e1eaa0dacf27ae352cc; no voters:
I20260808 18:33:55.718860 2054 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:55.718967 2059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:55.719280 2059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [term 1 LEADER]: Becoming Leader. State: Replica: 2f5cc1ee11294e1eaa0dacf27ae352cc, State: Running, Role: LEADER
I20260808 18:33:55.719656 2059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } }
I20260808 18:33:55.720227 2054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:55.721454 2060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } } }
I20260808 18:33:55.721602 2060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:55.721989 2061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5cc1ee11294e1eaa0dacf27ae352cc. Latest consensus state: current_term: 1 leader_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5cc1ee11294e1eaa0dacf27ae352cc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35063 } } }
I20260808 18:33:55.722067 2068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:55.722095 2061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:55.725397 2068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:55.731104 2068 catalog_manager.cc:1347] Generated new cluster ID: 3eb34d32fa57472bb66e567bf165472a
I20260808 18:33:55.731182 2068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:55.746513 2068 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:55.747536 2068 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:55.756081 2068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5cc1ee11294e1eaa0dacf27ae352cc: Generated new TSK 0
I20260808 18:33:55.757035 2068 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 18:33:55.789395 2085 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 18:33:55.789507 2085 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 18:33:55.790308 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:55.772394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.27.251.254:35063)
0808 18:33:55.772568 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:55.772574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:55.772775 (+ 201us) server_negotiation.cc:407] Connection header received
0808 18:33:55.772923 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:55.772926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:55.772995 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:55.773120 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:33:55.773345 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.774199 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.775187 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.775538 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.775643 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.775945 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:33:55.789858 (+ 13913us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 18:33:55.790159 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:33:55.790382 2057 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 18:33:55.790491 2084 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:55.772265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35063 (local address 127.0.0.1:40098)
0808 18:33:55.772678 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:55.772692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:55.772795 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:55.773126 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:55.773129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:55.773136 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:33:55.773232 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:55.773238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.774317 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.774320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:55.775064 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:55.775073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:55.775661 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:55.775664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:55.775739 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:55.775763 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:33:55.790098 (+ 14335us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 18:33:55.790277 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.251.254:35063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38949/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260808 18:33:55.791278 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 1981
2026-08-08T18:33:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 18:33:55.833830 28655 mini_oidc.cc:152] Starting JWKS server
I20260808 18:33:55.834784 28655 webserver.cc:460] Webserver started at https://127.0.0.1:45285/ using document root <none> and password file <none>
I20260808 18:33:55.834908 28655 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 18:33:55.835208 28655 webserver.cc:460] Webserver started at http://127.0.0.1:34311/ using document root <none> and password file <none>
2026-08-08T18:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:55Z Disabled control of system clock
I20260808 18:33:55.845714 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43147
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:43147
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 18:33:55.963070 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:55.963361 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:55.963402 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:55.967497 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:55.967571 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:55.967587 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:55.967604 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:55.967620 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:33:55.967638 2092 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:33:55.967651 2092 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45285/jwks.json
I20260808 18:33:55.972690 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:43147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:55.973807 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:55.975188 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:55.984568 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:55.984606 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:55.984615 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:55.984771 2092 server_base.cc:1034] running on GCE node
I20260808 18:33:55.985455 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:55.986594 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:55.987793 2092 hybrid_clock.cc:648] HybridClock initialized: now 1786214035987763 us; error 44 us; skew 500 ppm
I20260808 18:33:55.989765 2092 webserver.cc:460] Webserver started at http://127.27.251.254:35311/ using document root <none> and password file <none>
I20260808 18:33:55.990365 2092 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:55.990461 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:55.990718 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:55.992997 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "b431623252e54433b78d10ac54228275"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "bc0142f7c3c10ace48874e5ad794b9db"
server_key_iv: "245294e452e1b699ef2f185afd556d47"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.993599 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "b431623252e54433b78d10ac54228275"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "bc0142f7c3c10ace48874e5ad794b9db"
server_key_iv: "245294e452e1b699ef2f185afd556d47"
server_key_version: "encryptionkey@0"
I20260808 18:33:55.997649 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:33:56.000264 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:56.001677 2092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:56.001832 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "b431623252e54433b78d10ac54228275"
format_stamp: "Formatted at 2026-08-08 18:33:55 on dist-test-slave-fd7l"
server_key: "bc0142f7c3c10ace48874e5ad794b9db"
server_key_iv: "245294e452e1b699ef2f185afd556d47"
server_key_version: "encryptionkey@0"
I20260808 18:33:56.001964 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:56.021304 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:56.022173 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:56.022403 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:56.029682 2092 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:43147
I20260808 18:33:56.029692 2159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:43147 every 8 connection(s)
I20260808 18:33:56.030706 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:56.032078 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2092
I20260808 18:33:56.032256 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:56.032675 28655 external_mini_cluster.cc:1428] Setting key 962b68dde9eb20e462ad6470fdbe93f1
I20260808 18:33:56.034466 2160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:56.040197 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275: Bootstrap starting.
I20260808 18:33:56.042330 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:56.034163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.27.251.254:43147)
0808 18:33:56.034964 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:56.035003 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:56.035056 (+ 53us) server_negotiation.cc:407] Connection header received
0808 18:33:56.035793 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:56.035835 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:56.036272 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:56.036652 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:56.037093 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.038062 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.038964 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.039452 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.039647 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.040618 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:56.040623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:56.040636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:56.041017 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:56.041077 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:56.041097 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:56.041207 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:56.041326 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 18:33:56.041608 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":157,"threads_started":1}
I20260808 18:33:56.042678 2160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:56.043558 2160 log.cc:824] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:56.045446 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:56.045882 2160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275: No bootstrap required, opened a new log
I20260808 18:33:56.049870 2160 raft_consensus.cc:348] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } }
I20260808 18:33:56.050170 2160 raft_consensus.cc:374] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:56.050257 2160 raft_consensus.cc:729] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b431623252e54433b78d10ac54228275, State: Initialized, Role: FOLLOWER
I20260808 18:33:56.050673 2160 consensus_queue.cc:260] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } }
I20260808 18:33:56.050833 2160 raft_consensus.cc:388] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:56.050922 2160 raft_consensus.cc:482] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:56.051024 2160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:56.052014 2160 raft_consensus.cc:504] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } }
I20260808 18:33:56.052372 2160 leader_election.cc:302] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b431623252e54433b78d10ac54228275; no voters:
I20260808 18:33:56.052789 2160 leader_election.cc:288] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:56.052947 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:56.053195 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [term 1 LEADER]: Becoming Leader. State: Replica: b431623252e54433b78d10ac54228275, State: Running, Role: LEADER
I20260808 18:33:56.053586 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } }
I20260808 18:33:56.054107 2160 sys_catalog.cc:564] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:56.055323 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b431623252e54433b78d10ac54228275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } } }
I20260808 18:33:56.055469 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:56.055658 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b431623252e54433b78d10ac54228275. Latest consensus state: current_term: 1 leader_uuid: "b431623252e54433b78d10ac54228275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b431623252e54433b78d10ac54228275" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43147 } } }
I20260808 18:33:56.055830 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:56.055894 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:56.059319 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:56.065528 2176 catalog_manager.cc:1347] Generated new cluster ID: 4b6be730e0d846c798368e35329ebb66
I20260808 18:33:56.065611 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:56.099082 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:56.100387 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:56.111034 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b431623252e54433b78d10ac54228275: Generated new TSK 0
I20260808 18:33:56.111841 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:56.136442 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:56.130973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.27.251.254:43147)
0808 18:33:56.131108 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:56.131113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:56.131263 (+ 150us) server_negotiation.cc:407] Connection header received
0808 18:33:56.131398 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:56.131401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:56.131471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:56.131596 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:33:56.131906 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.132775 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.133662 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.134015 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.134127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.134445 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:33:56.136053 (+ 1608us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 18:33:56.136294 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:33:56.136801 2164 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 18:33:56.138055 2192 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:56.130873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:43147 (local address 127.0.0.1:58572)
0808 18:33:56.131157 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:56.131171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:56.131278 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:56.131597 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:56.131599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:56.131605 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:33:56.131723 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:56.131728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.132925 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.132928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:56.133546 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:56.133554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.134161 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.134164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:56.134231 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.134254 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:33:56.136218 (+ 1964us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 18:33:56.137872 (+ 1654us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.251.254:43147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45285/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1456,"thread_start_us":106,"threads_started":1}
I20260808 18:33:56.138651 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2092
E20260808 18:33:56.147858 2193 webserver.cc:533] Webserver: sslize failed:
E20260808 18:33:56.147936 2193 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T18:33:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T18:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:56Z Disabled control of system clock
I20260808 18:33:56.191699 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45787
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45787
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/testchainca.cert with env {}
W20260808 18:33:56.307816 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:56.308084 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:56.308168 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:56.312219 2197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:56.312295 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:56.312311 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:56.312328 2197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:56.312395 2197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 18:33:56.312424 2197 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 18:33:56.312475 2197 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 18:33:56.318275 2197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:56.319460 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:56.320570 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:56.329324 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:56.329344 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:56.329370 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:56.329702 2197 server_base.cc:1034] running on GCE node
I20260808 18:33:56.330351 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:56.331562 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:56.332792 2197 hybrid_clock.cc:648] HybridClock initialized: now 1786214036332754 us; error 49 us; skew 500 ppm
I20260808 18:33:56.334803 2197 webserver.cc:460] Webserver started at http://127.27.251.254:45691/ using document root <none> and password file <none>
I20260808 18:33:56.335438 2197 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:56.335536 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:56.335774 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:56.337934 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "a10630c7900243e680771a8067ce0757"
format_stamp: "Formatted at 2026-08-08 18:33:56 on dist-test-slave-fd7l"
server_key: "3525f56c85b37ecee719598e217862bc"
server_key_iv: "f8f0815aff5ec211831136f83e566e22"
server_key_version: "encryptionkey@0"
I20260808 18:33:56.338512 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "a10630c7900243e680771a8067ce0757"
format_stamp: "Formatted at 2026-08-08 18:33:56 on dist-test-slave-fd7l"
server_key: "3525f56c85b37ecee719598e217862bc"
server_key_iv: "f8f0815aff5ec211831136f83e566e22"
server_key_version: "encryptionkey@0"
I20260808 18:33:56.342227 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:33:56.344702 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:56.345971 2197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:56.346140 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a10630c7900243e680771a8067ce0757"
format_stamp: "Formatted at 2026-08-08 18:33:56 on dist-test-slave-fd7l"
server_key: "3525f56c85b37ecee719598e217862bc"
server_key_iv: "f8f0815aff5ec211831136f83e566e22"
server_key_version: "encryptionkey@0"
I20260808 18:33:56.346275 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:56.359823 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:56.360589 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:56.360801 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:56.367733 2197 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45787
I20260808 18:33:56.367761 2267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45787 every 8 connection(s)
I20260808 18:33:56.368858 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:56.371857 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:56.377581 2268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757: Bootstrap starting.
I20260808 18:33:56.378340 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2197
I20260808 18:33:56.378444 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:56.378784 28655 external_mini_cluster.cc:1428] Setting key 1f0fdf46af9954e4cd3373a40b524896
I20260808 18:33:56.380455 2268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:56.381321 2268 log.cc:824] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757: Log is configured to *not* fsync() on all Append() calls
I20260808 18:33:56.383430 2268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757: No bootstrap required, opened a new log
I20260808 18:33:56.387250 2268 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } }
I20260808 18:33:56.387540 2268 raft_consensus.cc:374] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:56.387619 2268 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10630c7900243e680771a8067ce0757, State: Initialized, Role: FOLLOWER
I20260808 18:33:56.388111 2268 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } }
I20260808 18:33:56.388268 2268 raft_consensus.cc:388] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:56.388377 2268 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:56.388500 2268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:56.388697 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:56.380065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.27.251.254:45787)
0808 18:33:56.380809 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:56.380829 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:56.380866 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:33:56.381566 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:56.381591 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:56.382052 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:56.382402 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:56.382847 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.383914 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.384894 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.385405 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.385590 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.386569 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:56.386574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:56.386587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:33:56.386977 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:56.387033 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:33:56.387056 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:56.387170 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:56.387324 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 18:33:56.387642 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":132,"threads_started":1}
I20260808 18:33:56.389575 2268 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } }
I20260808 18:33:56.389969 2268 leader_election.cc:302] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a10630c7900243e680771a8067ce0757; no voters:
I20260808 18:33:56.390354 2268 leader_election.cc:288] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:56.390470 2274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:56.390712 2274 raft_consensus.cc:686] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [term 1 LEADER]: Becoming Leader. State: Replica: a10630c7900243e680771a8067ce0757, State: Running, Role: LEADER
I20260808 18:33:56.391106 2274 consensus_queue.cc:237] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } }
I20260808 18:33:56.391661 2268 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:56.391795 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:33:56.392277 2274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a10630c7900243e680771a8067ce0757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } } }
I20260808 18:33:56.392387 2274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:56.392620 2274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a10630c7900243e680771a8067ce0757. Latest consensus state: current_term: 1 leader_uuid: "a10630c7900243e680771a8067ce0757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10630c7900243e680771a8067ce0757" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45787 } } }
I20260808 18:33:56.392769 2274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:56.393111 2281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:56.396950 2281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:56.403389 2281 catalog_manager.cc:1347] Generated new cluster ID: b768ed2b5b61466ea811982c5544def7
I20260808 18:33:56.403527 2281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:56.437592 2281 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:56.438634 2281 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:56.451329 2281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a10630c7900243e680771a8067ce0757: Generated new TSK 0
I20260808 18:33:56.452093 2281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:33:56.484072 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:56.462609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.27.251.254:45787)
0808 18:33:56.462742 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:56.462747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:56.462998 (+ 251us) server_negotiation.cc:407] Connection header received
0808 18:33:56.463164 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:56.463166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:56.463215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:56.463304 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0808 18:33:56.463578 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.464098 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.465372 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.465608 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.465688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.466035 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 18:33:56.483662 (+ 17627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 18:33:56.483945 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40786: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 18:33:56.484194 2272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40786: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 18:33:56.484365 2300 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:56.462585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45787 (local address 127.0.0.1:40786)
0808 18:33:56.462900 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:56.462915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:56.463033 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:56.463333 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:56.463336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:56.463345 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 18:33:56.463465 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:56.463472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.464224 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.464228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:56.465255 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:56.465267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:56.465732 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:56.465736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:56.465832 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:56.465859 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 18:33:56.483922 (+ 18063us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 18:33:56.484174 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.251.254:45787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260808 18:33:56.485064 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2197
2026-08-08T18:33:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (325 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (136 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:56 dist-test-slave-fd7l krb5kdc[2313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:56 dist-test-slave-fd7l krb5kdc[2313](info): set up 2 sockets
Aug 08 18:33:56 dist-test-slave-fd7l krb5kdc[2313](info): commencing operation
krb5kdc: starting...
W20260808 18:33:57.883168 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:33:57 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:33:58.013868 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42259
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:58.131001 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:58.131328 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:58.131410 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:58.135422 2327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:33:58.135520 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:58.135557 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:58.135594 2327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:33:58.135629 2327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:33:58.140904 2327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:42259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:58.141983 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:58.143050 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:58.151902 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.151921 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.151882 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:58.152292 2327 server_base.cc:1034] running on GCE node
I20260808 18:33:58.152859 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:58.154017 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:58.155174 2327 hybrid_clock.cc:648] HybridClock initialized: now 1786214038155155 us; error 30 us; skew 500 ppm
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:58.158211 2327 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:33:58.159206 2327 webserver.cc:460] Webserver started at http://127.27.251.254:38881/ using document root <none> and password file <none>
I20260808 18:33:58.159834 2327 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:58.159899 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:58.160156 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:58.162367 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "93e491d669f24f6b89b9d8a943953283"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "0e36811fd063d2e5a760614f4e3f2b99"
server_key_iv: "faa399aebec8d6593f7330345b9d480b"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.162982 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "93e491d669f24f6b89b9d8a943953283"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "0e36811fd063d2e5a760614f4e3f2b99"
server_key_iv: "faa399aebec8d6593f7330345b9d480b"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.166994 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:33:58.169608 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.170933 2327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:33:58.171069 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "93e491d669f24f6b89b9d8a943953283"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "0e36811fd063d2e5a760614f4e3f2b99"
server_key_iv: "faa399aebec8d6593f7330345b9d480b"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.171181 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:58.194036 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:58.197758 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:58.198000 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:58.205932 2327 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:42259
I20260808 18:33:58.205947 2396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:42259 every 8 connection(s)
I20260808 18:33:58.206954 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:33:58.209760 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:33:58.210651 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2327
I20260808 18:33:58.210783 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:33:58.211114 28655 external_mini_cluster.cc:1428] Setting key 241cab35fa49f8cf8d4a4b65641501b3
I20260808 18:33:58.216661 2397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: Bootstrap starting.
I20260808 18:33:58.219285 2397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: Neither blocks nor log segments found. Creating new log.
I20260808 18:33:58.220093 2397 log.cc:824] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:58.222385 2397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: No bootstrap required, opened a new log
I20260808 18:33:58.225852 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.212650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.27.251.254:42259)
0808 18:33:58.213061 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:58.213080 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:58.213125 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:33:58.213937 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:58.213961 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:58.214393 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:58.214778 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:58.215488 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.216975 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.217848 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.218503 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.218679 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.221368 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:58.221373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:58.221385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:58.221411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:58.223144 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.223590 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.223595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.223600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.223709 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.223992 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.223996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.223997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.224371 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:58.224612 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:58.224876 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 18:33:58.225147 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260808 18:33:58.226110 2397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } }
I20260808 18:33:58.226351 2397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:33:58.226425 2397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e491d669f24f6b89b9d8a943953283, State: Initialized, Role: FOLLOWER
I20260808 18:33:58.226881 2397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } }
I20260808 18:33:58.227042 2397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:33:58.227109 2397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:33:58.227206 2397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:33:58.228295 2397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } }
I20260808 18:33:58.228744 2397 leader_election.cc:302] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e491d669f24f6b89b9d8a943953283; no voters:
I20260808 18:33:58.229171 2397 leader_election.cc:288] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:33:58.229285 2402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:33:58.229537 2402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [term 1 LEADER]: Becoming Leader. State: Replica: 93e491d669f24f6b89b9d8a943953283, State: Running, Role: LEADER
I20260808 18:33:58.229874 2402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } }
I20260808 18:33:58.230629 2397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:33:58.231551 2404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e491d669f24f6b89b9d8a943953283. Latest consensus state: current_term: 1 leader_uuid: "93e491d669f24f6b89b9d8a943953283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } } }
I20260808 18:33:58.231688 2404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [sys.catalog]: This master's current role is: LEADER
I20260808 18:33:58.233404 2403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e491d669f24f6b89b9d8a943953283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e491d669f24f6b89b9d8a943953283" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42259 } } }
I20260808 18:33:58.233548 2403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283 [sys.catalog]: This master's current role is: LEADER
W20260808 18:33:58.236196 2419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:33:58.236279 2419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:33:58.236380 2413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:33:58.237370 2413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:33:58.242681 2413 catalog_manager.cc:1347] Generated new cluster ID: d96ff8c4ded84350b99f4101db7b329a
I20260808 18:33:58.242745 2413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:33:58.262036 2413 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:33:58.262982 2413 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:33:58.277668 2413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93e491d669f24f6b89b9d8a943953283: Generated new TSK 0
I20260808 18:33:58.278489 2413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:33:58.319695 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:58.435948 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:58.436210 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:58.436254 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:58.440176 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:58.440239 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:58.440297 2425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:33:58.445480 2425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:58.446578 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:58.447556 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:58.457721 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.457854 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.457744 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:58.457974 2425 server_base.cc:1034] running on GCE node
I20260808 18:33:58.458516 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:58.459290 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:58.460481 2425 hybrid_clock.cc:648] HybridClock initialized: now 1786214038460464 us; error 26 us; skew 500 ppm
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:58.463667 2425 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:33:58.464656 2425 webserver.cc:460] Webserver started at http://127.27.251.193:46319/ using document root <none> and password file <none>
I20260808 18:33:58.465238 2425 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:58.465301 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:58.465546 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:58.467675 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "d4cff078af994d1db0ef6e3d65a9836b"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "ea97278951547db5734726116c1c197d"
server_key_iv: "0c4f3bd92c27835ce802439eca1ed5dc"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.468266 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "d4cff078af994d1db0ef6e3d65a9836b"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "ea97278951547db5734726116c1c197d"
server_key_iv: "0c4f3bd92c27835ce802439eca1ed5dc"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.472201 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 18:33:58.475042 2441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.476445 2425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:33:58.476615 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "d4cff078af994d1db0ef6e3d65a9836b"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "ea97278951547db5734726116c1c197d"
server_key_iv: "0c4f3bd92c27835ce802439eca1ed5dc"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.476753 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:58.491086 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:58.494172 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:58.494383 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:58.495009 2425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:58.495869 2425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:58.495942 2425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.496016 2425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:58.496059 2425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.505584 2425 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:39307
I20260808 18:33:58.505630 2561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:39307 every 8 connection(s)
I20260808 18:33:58.506537 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:33:58.506964 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2425
I20260808 18:33:58.507128 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:33:58.507594 28655 external_mini_cluster.cc:1428] Setting key c0bd0da37b7e579f596d0c3b46363357
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:33:58.521871 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.508458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:58777 (local address 127.27.251.254:42259)
0808 18:33:58.508595 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:58.508600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:58.509613 (+ 1013us) server_negotiation.cc:407] Connection header received
0808 18:33:58.511774 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:58.511777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:58.511833 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:58.511913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:58.513237 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.513782 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.515060 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.515311 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.515428 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.518047 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:58.518051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:58.518055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:58.518087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:58.519588 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.520176 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.520179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.520180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.520268 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.520695 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.520699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.520700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.520925 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:58.521041 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:58.521557 (+ 516us) server_negotiation.cc:299] Negotiation successful
0808 18:33:58.521687 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:33:58.522724 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.508909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42259 (local address 127.27.251.193:58777)
0808 18:33:58.509422 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:58.509495 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:58.511505 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:58.512101 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:58.512109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:58.512502 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:58.513078 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.513092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.513927 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.513930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.514926 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.514937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.515482 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.515487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.515658 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.516322 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:58.516339 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:58.517846 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:58.519713 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.519719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.519734 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.519998 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.520366 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.520370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.520372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.520591 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.521050 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:58.521060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:58.521309 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 18:33:58.521496 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 18:33:58.521831 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260808 18:33:58.524070 2562 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42259
I20260808 18:33:58.524339 2562 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:58.525017 2562 heartbeater.cc:511] Master 127.27.251.254:42259 requested a full tablet report, sending...
I20260808 18:33:58.526681 2360 ts_manager.cc:194] Registered new tserver with Master: d4cff078af994d1db0ef6e3d65a9836b (127.27.251.193:39307)
I20260808 18:33:58.527803 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:58777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:33:58.539788 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:58.653884 2567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:58.654192 2567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:58.654284 2567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:58.658309 2567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:58.658360 2567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:58.658416 2567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:33:58.663471 2567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:58.664417 2567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:58.665463 2567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:58.675086 2576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.675122 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.675133 2573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:58.675540 2567 server_base.cc:1034] running on GCE node
I20260808 18:33:58.675948 2567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:58.676640 2567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:58.677832 2567 hybrid_clock.cc:648] HybridClock initialized: now 1786214038677817 us; error 25 us; skew 500 ppm
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:58.680624 2567 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:33:58.681591 2567 webserver.cc:460] Webserver started at http://127.27.251.194:40709/ using document root <none> and password file <none>
I20260808 18:33:58.682194 2567 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:58.682291 2567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:58.682530 2567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:58.684628 2567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "3ac2b715d4c04640a8ef2e8e73a908b4"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "57ab4319ba1a7435e1992421dc4197d7"
server_key_iv: "56eada34e868c5b3857029a4753a1a87"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.685228 2567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "3ac2b715d4c04640a8ef2e8e73a908b4"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "57ab4319ba1a7435e1992421dc4197d7"
server_key_iv: "56eada34e868c5b3857029a4753a1a87"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.689073 2567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:33:58.691587 2583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.692963 2567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:33:58.693101 2567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "3ac2b715d4c04640a8ef2e8e73a908b4"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "57ab4319ba1a7435e1992421dc4197d7"
server_key_iv: "56eada34e868c5b3857029a4753a1a87"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.693224 2567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:58.726498 2567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:58.729703 2567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:58.729930 2567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:58.730566 2567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:58.731491 2567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:58.731598 2567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.731680 2567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:58.731736 2567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.741387 2567 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:44177
I20260808 18:33:58.741413 2700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:44177 every 8 connection(s)
I20260808 18:33:58.742319 2567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:33:58.747329 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2567
I20260808 18:33:58.747458 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:33:58.747781 28655 external_mini_cluster.cc:1428] Setting key 7d81693390305e1fcbb30e0bf66bbdfd
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:58.756953 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.744095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42947 (local address 127.27.251.254:42259)
0808 18:33:58.744235 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:58.744239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:58.745140 (+ 901us) server_negotiation.cc:407] Connection header received
0808 18:33:58.746483 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:58.746486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:58.746535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:58.746610 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:58.748079 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.748697 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.750031 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.750443 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.750568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.753353 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:58.753356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:58.753358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:58.753381 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:58.754813 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.755379 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.755385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.755387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.755473 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.755840 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.755844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.755845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.756031 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:58.756148 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:58.756669 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 18:33:58.756791 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:58.757843 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.744432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42259 (local address 127.27.251.194:42947)
0808 18:33:58.744954 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:58.744997 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:58.746288 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:58.746778 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:58.746784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:58.747241 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:58.747928 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.747949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.748833 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.748837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.749900 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.749910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.750624 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.750627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.750763 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.751395 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:58.751423 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:58.753177 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:58.754938 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.754945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.754962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.755267 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.755578 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.755581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.755583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.755737 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.756153 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:58.756162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:58.756406 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:33:58.756623 (+ 217us) client_negotiation.cc:240] Negotiation successful
0808 18:33:58.756950 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:33:58.759006 2701 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42259
I20260808 18:33:58.759268 2701 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:58.759812 2701 heartbeater.cc:511] Master 127.27.251.254:42259 requested a full tablet report, sending...
I20260808 18:33:58.760905 2360 ts_manager.cc:194] Registered new tserver with Master: 3ac2b715d4c04640a8ef2e8e73a908b4 (127.27.251.194:44177)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:33:58.761588 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:42947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:33:58.777523 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:33:58.891896 2709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:33:58.892131 2709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:33:58.892171 2709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:33:58.896368 2709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:33:58.896421 2709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:33:58.896559 2709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:33:58.901988 2709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:33:58.903066 2709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:33:58.904167 2709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:33:58.913815 2714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.913831 2715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:33:58.913868 2718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:33:58.914130 2709 server_base.cc:1034] running on GCE node
I20260808 18:33:58.914546 2709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:33:58.915169 2709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:33:58.916358 2709 hybrid_clock.cc:648] HybridClock initialized: now 1786214038916299 us; error 84 us; skew 500 ppm
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:58.919102 2709 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:33:58.919934 2709 webserver.cc:460] Webserver started at http://127.27.251.195:38449/ using document root <none> and password file <none>
I20260808 18:33:58.920492 2709 fs_manager.cc:359] Metadata directory not provided
I20260808 18:33:58.920657 2709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:33:58.920899 2709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:33:58.922961 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "8b4656f4dadf41158bc3c0bc40683ce7"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "349d755d1eee311650c31df6b71866c1"
server_key_iv: "d335a0b38173115a3f8c67480a56d389"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.923530 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "8b4656f4dadf41158bc3c0bc40683ce7"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "349d755d1eee311650c31df6b71866c1"
server_key_iv: "d335a0b38173115a3f8c67480a56d389"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.927197 2709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260808 18:33:58.929637 2725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.931002 2709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:33:58.931149 2709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "8b4656f4dadf41158bc3c0bc40683ce7"
format_stamp: "Formatted at 2026-08-08 18:33:58 on dist-test-slave-fd7l"
server_key: "349d755d1eee311650c31df6b71866c1"
server_key_iv: "d335a0b38173115a3f8c67480a56d389"
server_key_version: "encryptionkey@0"
I20260808 18:33:58.931250 2709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:33:58.947309 2709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:33:58.951196 2709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:33:58.951359 2709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:33:58.951936 2709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:33:58.952806 2709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:33:58.952857 2709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.952898 2709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:33:58.952914 2709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:33:58.961858 2709 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:38123
I20260808 18:33:58.961943 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:38123 every 8 connection(s)
I20260808 18:33:58.962832 2709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:33:58.964725 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2709
I20260808 18:33:58.964860 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:33:58.965233 28655 external_mini_cluster.cc:1428] Setting key 1eb75f7734c41b3c7ae937dc9d324ceb
Aug 08 18:33:58 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214038, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:33:58.977855 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.964828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:58089 (local address 127.27.251.254:42259)
0808 18:33:58.964958 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:58.964964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:58.966201 (+ 1237us) server_negotiation.cc:407] Connection header received
0808 18:33:58.968015 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:58.968017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:58.968074 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:58.968148 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:58.969433 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.970034 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.971196 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.971416 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.971514 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.974154 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:58.974157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:58.974160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:58.974188 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:58.975774 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.976309 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.976312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.976313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.976398 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:58.976811 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:58.976815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:58.976816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:58.977018 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:58.977120 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:58.977611 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 18:33:58.977720 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 18:33:58.978972 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:58.965379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42259 (local address 127.27.251.195:58089)
0808 18:33:58.966024 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0808 18:33:58.966084 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:33:58.967806 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:33:58.968328 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:33:58.968335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:33:58.968750 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:33:58.969280 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.969295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.970155 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.970157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.971057 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:33:58.971066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:33:58.971565 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:33:58.971568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:33:58.971724 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:58.972333 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:33:58.972349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:33:58.973987 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:33:58.975912 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.975917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.975930 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.976195 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.976500 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:33:58.976502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:33:58.976503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:33:58.976698 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:33:58.977128 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:33:58.977134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:33:58.977374 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 18:33:58.977528 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 18:33:58.977822 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":215,"threads_started":1}
I20260808 18:33:58.980270 2844 heartbeater.cc:347] Connected to a master server at 127.27.251.254:42259
I20260808 18:33:58.980481 2844 heartbeater.cc:464] Registering TS with master...
I20260808 18:33:58.981042 2844 heartbeater.cc:511] Master 127.27.251.254:42259 requested a full tablet report, sending...
I20260808 18:33:58.982102 2360 ts_manager.cc:194] Registered new tserver with Master: 8b4656f4dadf41158bc3c0bc40683ce7 (127.27.251.195:38123)
I20260808 18:33:58.982702 2360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:58089
I20260808 18:33:58.991549 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/test-admin.keytab.
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.009851 28655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 18:33:59.012861 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:59.011212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.27.251.193:39307)
0808 18:33:59.011491 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.011505 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.011555 (+ 50us) server_negotiation.cc:407] Connection header received
0808 18:33:59.011607 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.011612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.011799 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.011922 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.012627 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43612: BlockingRecv error: recv got EOF from 127.0.0.1:43612 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
W20260808 18:33:59.012899 28655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.014107 28655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.015086 28655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:59.024785 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:59.015888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.27.251.193:39307)
0808 18:33:59.016018 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.016021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.016115 (+ 94us) server_negotiation.cc:407] Connection header received
0808 18:33:59.016221 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.016223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.016262 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.016327 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.016968 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.017806 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.018600 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.018974 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.019062 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:59.021502 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:59.021511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:59.021524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:59.021548 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:59.023062 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.023509 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.023515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.023516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.023598 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.023828 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.023830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.023831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.024061 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:59.024207 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:59.024414 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 18:33:59.024664 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 18:33:59.028378 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:59.027339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.27.251.193:39307)
0808 18:33:59.027485 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.027489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.027578 (+ 89us) server_negotiation.cc:407] Connection header received
0808 18:33:59.027732 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.027734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.027778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.027856 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.028261 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43642: BlockingRecv error: recv got EOF from 127.0.0.1:43642 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 18:33:59.028671 28655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.029685 28655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.030660 28655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:59.039057 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:59.031552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.27.251.193:39307)
0808 18:33:59.031654 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.031658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.031754 (+ 96us) server_negotiation.cc:407] Connection header received
0808 18:33:59.031828 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.031830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.031862 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.031923 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.032403 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.033005 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.033740 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.033957 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.034091 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:59.036257 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:59.036266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:59.036268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:59.036289 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:59.037491 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.037913 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.037916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.037918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.037999 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.038273 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.038277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.038279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.038437 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:59.038570 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:59.038810 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 18:33:59.038921 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 18:33:59.041522 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:33:59.040674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.27.251.193:39307)
0808 18:33:59.040777 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.040781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.040911 (+ 130us) server_negotiation.cc:407] Connection header received
0808 18:33:59.040990 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.040992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.041023 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.041086 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.041408 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43648: BlockingRecv error: recv got EOF from 127.0.0.1:43648 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 18:33:59.041726 28655 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786214007390461-28655-0/krb5kdc/krb5cc' of type FILE
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.042708 28655 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:33:59.043565 28655 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:33:59.051437 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:33:59.044333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.27.251.193:39307)
0808 18:33:59.044452 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:33:59.044456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:33:59.044685 (+ 229us) server_negotiation.cc:407] Connection header received
0808 18:33:59.044757 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:33:59.044759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:33:59.044791 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:33:59.044853 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:33:59.045287 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.045774 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.046490 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:33:59.046714 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:33:59.046789 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:33:59.048899 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:33:59.048902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:33:59.048904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:33:59.048925 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:33:59.050127 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.050502 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.050505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.050506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.050577 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:33:59.050811 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:33:59.050813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:33:59.050814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:33:59.050964 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:33:59.051074 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:33:59.051243 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 18:33:59.051329 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:33:59.052637 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2425
I20260808 18:33:59.061398 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2567
I20260808 18:33:59.070411 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2709
I20260808 18:33:59.078234 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2327
2026-08-08T18:33:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2329 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2867](info): set up 2 sockets
Aug 08 18:33:59 dist-test-slave-fd7l krb5kdc[2867](info): commencing operation
krb5kdc: starting...
W20260808 18:34:00.167686 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:00 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab.
Entry for principal oryx/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab.
I20260808 18:34:00.303321 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36769
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab
--principal=oryx/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:00.417708 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:00.417980 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:00.418053 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:00.422108 2884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:00.422190 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:00.422211 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:00.422228 2884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:00.422271 2884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:00.427594 2884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab
--principal=oryx/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:00.428776 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:00.430168 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:00.439245 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:00.439258 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:00.439288 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:00.439608 2884 server_base.cc:1034] running on GCE node
I20260808 18:34:00.440284 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:00.441469 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:00.442677 2884 hybrid_clock.cc:648] HybridClock initialized: now 1786214040442638 us; error 49 us; skew 500 ppm
Aug 08 18:34:00 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:00.445634 2884 init.cc:377] Logged in from keytab as oryx/127.27.251.254@KRBTEST.COM (short username oryx)
I20260808 18:34:00.446628 2884 webserver.cc:460] Webserver started at http://127.27.251.254:42671/ using document root <none> and password file <none>
I20260808 18:34:00.447211 2884 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:00.447306 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:00.447553 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:00.449734 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "f224c5b1dd8e45a48424e6c437279130"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "f3d6060051e25d9ee3279409abeea573"
server_key_iv: "6e3d6668b9bfdca00556dd4415294bf8"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.450299 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "f224c5b1dd8e45a48424e6c437279130"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "f3d6060051e25d9ee3279409abeea573"
server_key_iv: "6e3d6668b9bfdca00556dd4415294bf8"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.454247 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:00.456785 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:00.458106 2884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:00.458261 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "f224c5b1dd8e45a48424e6c437279130"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "f3d6060051e25d9ee3279409abeea573"
server_key_iv: "6e3d6668b9bfdca00556dd4415294bf8"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.458384 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:00.505352 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:00.508456 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:00.508721 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:00.516055 2884 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:36769
I20260808 18:34:00.516355 2954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:36769 every 8 connection(s)
I20260808 18:34:00.517155 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:00.519990 2955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:00.521099 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2884
I20260808 18:34:00.521293 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:00.521677 28655 external_mini_cluster.cc:1428] Setting key d9fc2c2a7bc877b4c90dbe2381c48f59
I20260808 18:34:00.527103 2955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Bootstrap starting.
I20260808 18:34:00.529812 2955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:00.530618 2955 log.cc:824] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:34:00 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:00.533031 2955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: No bootstrap required, opened a new log
I20260808 18:34:00.537498 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:00.523191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.27.251.254:36769)
0808 18:34:00.523620 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:00.523654 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:00.523703 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:34:00.524551 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:00.524575 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:00.524999 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:00.525371 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:00.526178 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.527577 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.528433 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.529153 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.529311 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:00.532107 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:00.532115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:00.532132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:00.532164 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:00.534236 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:00.534676 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:00.534682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:00.534689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:00.534836 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:00.535107 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:00.535110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:00.535112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:00.535717 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:00.535913 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:00.536159 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 18:34:00.536462 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260808 18:34:00.537751 2955 raft_consensus.cc:348] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } }
I20260808 18:34:00.538026 2955 raft_consensus.cc:374] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:00.538087 2955 raft_consensus.cc:729] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f224c5b1dd8e45a48424e6c437279130, State: Initialized, Role: FOLLOWER
I20260808 18:34:00.538569 2955 consensus_queue.cc:260] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } }
I20260808 18:34:00.538736 2955 raft_consensus.cc:388] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:00.538807 2955 raft_consensus.cc:482] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:00.538902 2955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:00.540037 2955 raft_consensus.cc:504] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } }
I20260808 18:34:00.540395 2955 leader_election.cc:302] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f224c5b1dd8e45a48424e6c437279130; no voters:
I20260808 18:34:00.540737 2955 leader_election.cc:288] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:00.540874 2961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:00.541144 2961 raft_consensus.cc:686] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 LEADER]: Becoming Leader. State: Replica: f224c5b1dd8e45a48424e6c437279130, State: Running, Role: LEADER
I20260808 18:34:00.541462 2961 consensus_queue.cc:237] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } }
I20260808 18:34:00.542001 2955 sys_catalog.cc:564] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:00.543237 2962 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } }
I20260808 18:34:00.543391 2962 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:00.543237 2963 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f224c5b1dd8e45a48424e6c437279130. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } }
I20260808 18:34:00.543495 2963 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:00.543843 2972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:00.547153 2972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:00.553399 2972 catalog_manager.cc:1347] Generated new cluster ID: f6d51657d34645b794c56882bf5bec93
I20260808 18:34:00.553484 2972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:00.582865 2972 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:00.583823 2972 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:00.589911 2972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Generated new TSK 0
I20260808 18:34:00.590603 2972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
I20260808 18:34:00.628172 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:00.764353 2985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:00.764688 2985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:00.764772 2985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:00.768833 2985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:00.768893 2985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:00.768985 2985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:00.774142 2985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:00.775194 2985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:00.776224 2985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:00.786381 2991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:00.786389 2995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:00.786432 2992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:00.786470 2985 server_base.cc:1034] running on GCE node
I20260808 18:34:00.787038 2985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:00.787711 2985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:00.788909 2985 hybrid_clock.cc:648] HybridClock initialized: now 1786214040788887 us; error 35 us; skew 500 ppm
Aug 08 18:34:00 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:00.791702 2985 init.cc:377] Logged in from keytab as oryx/127.27.251.193@KRBTEST.COM (short username oryx)
I20260808 18:34:00.792629 2985 webserver.cc:460] Webserver started at http://127.27.251.193:45695/ using document root <none> and password file <none>
I20260808 18:34:00.793246 2985 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:00.793337 2985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:00.793550 2985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:00.795562 2985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "bf727fe62da441c98127b57c40878529"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "fc135280b6a10cc365bfcdfedb3b479c"
server_key_iv: "6832a2ce5da4c60c535a401e027abef7"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.796136 2985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "bf727fe62da441c98127b57c40878529"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "fc135280b6a10cc365bfcdfedb3b479c"
server_key_iv: "6832a2ce5da4c60c535a401e027abef7"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.799786 2985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:34:00.802274 3003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:00.803663 2985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:00.803817 2985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "bf727fe62da441c98127b57c40878529"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "fc135280b6a10cc365bfcdfedb3b479c"
server_key_iv: "6832a2ce5da4c60c535a401e027abef7"
server_key_version: "encryptionkey@0"
I20260808 18:34:00.803937 2985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:00.831686 2985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:00.834992 2985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:00.835211 2985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:00.835867 2985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:00.836792 2985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:00.836863 2985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:00.836935 2985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:00.836977 2985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:00.846722 2985 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:36307
I20260808 18:34:00.846781 3124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:36307 every 8 connection(s)
I20260808 18:34:00.847716 2985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:00.848214 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 2985
I20260808 18:34:00.848347 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:00.848738 28655 external_mini_cluster.cc:1428] Setting key d63978aa9c8b26e94f95e7d4f1116db6
Aug 08 18:34:00 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.194@KRBTEST.COM" created.
I20260808 18:34:00.862655 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:00.849787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:42383 (local address 127.27.251.254:36769)
0808 18:34:00.849926 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:00.849931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:00.851003 (+ 1072us) server_negotiation.cc:407] Connection header received
0808 18:34:00.852411 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:00.852414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:00.852467 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:00.852608 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:00.853815 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.854353 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.855511 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.855752 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.855858 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:00.858660 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:00.858663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:00.858671 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:00.858696 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:00.860160 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:00.860815 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:00.860818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:00.860819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:00.860899 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:00.861291 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:00.861297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:00.861302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:00.861527 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:00.861608 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:00.862323 (+ 715us) server_negotiation.cc:299] Negotiation successful
0808 18:34:00.862467 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:34:00.863685 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:00.850326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:42383)
0808 18:34:00.850860 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:00.850903 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:00.852207 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:00.852794 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:00.852801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:00.853174 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:00.853667 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:00.853681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.854487 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.854490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:00.855379 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:00.855389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:00.855912 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:00.855917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:00.856088 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:00.856798 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:00.856815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:00.858492 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:00.860287 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:00.860292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:00.860304 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:00.860723 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:00.861008 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:00.861010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:00.861011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:00.861157 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:00.861634 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:00.861639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:00.861969 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 18:34:00.862278 (+ 309us) client_negotiation.cc:240] Negotiation successful
0808 18:34:00.862675 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":138,"threads_started":1}
I20260808 18:34:00.865190 3125 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:00.865460 3125 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:00.865981 3125 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
I20260808 18:34:00.867761 2915 ts_manager.cc:194] Registered new tserver with Master: bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:00.869062 2915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:42383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
I20260808 18:34:00.879827 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:00.993227 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:00.993532 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:00.993604 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:00.997610 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:00.997690 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:00.997778 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:01.003072 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:01.004088 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:01.005098 3130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:01.014969 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.014956 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.015003 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:01.015127 3130 server_base.cc:1034] running on GCE node
I20260808 18:34:01.015724 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:01.016386 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:01.017589 3130 hybrid_clock.cc:648] HybridClock initialized: now 1786214041017574 us; error 24 us; skew 500 ppm
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:01.020344 3130 init.cc:377] Logged in from keytab as oryx/127.27.251.194@KRBTEST.COM (short username oryx)
I20260808 18:34:01.021265 3130 webserver.cc:460] Webserver started at http://127.27.251.194:36709/ using document root <none> and password file <none>
I20260808 18:34:01.021842 3130 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.021935 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.022187 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:01.024197 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "7b2064f1f9d04afe894fa263e6309409"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "44e20cf4a68352658767d26ec4a0fa9a"
server_key_iv: "9e44e2eed3478b575d5956cf7253cb2a"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.024791 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "7b2064f1f9d04afe894fa263e6309409"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "44e20cf4a68352658767d26ec4a0fa9a"
server_key_iv: "9e44e2eed3478b575d5956cf7253cb2a"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.028455 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:34:01.030819 3146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.031963 3130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:34:01.032109 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "7b2064f1f9d04afe894fa263e6309409"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "44e20cf4a68352658767d26ec4a0fa9a"
server_key_iv: "9e44e2eed3478b575d5956cf7253cb2a"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.032222 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.053843 3130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:01.057029 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:01.057242 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:01.057854 3130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:01.058694 3130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:01.058769 3130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.058843 3130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:01.058897 3130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.068651 3130 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:33077
I20260808 18:34:01.068679 3268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:33077 every 8 connection(s)
I20260808 18:34:01.069594 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:01.077150 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3130
I20260808 18:34:01.077284 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:01.077591 28655 external_mini_cluster.cc:1428] Setting key 6ec826de8ca9784fad4df844ee8ad0b0
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:01.084015 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.071461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:49073 (local address 127.27.251.254:36769)
0808 18:34:01.071594 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.071598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.072412 (+ 814us) server_negotiation.cc:407] Connection header received
0808 18:34:01.073916 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.073920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.073983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.074093 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.075298 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.075833 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.077137 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.077456 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.077571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.080180 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.080186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.080193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.080215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.081791 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.082345 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.082350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.082352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.082439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.082798 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.082802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.082804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.082984 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.083097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.083706 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.083841 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:34:01.084903 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.071790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:49073)
0808 18:34:01.072251 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.072296 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.073696 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.074287 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.074294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.074655 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.075147 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.075161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.075967 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.075970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.077000 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.077011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.077618 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.077623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.077811 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.078479 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.078496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.080001 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.081917 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.081923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.081937 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.082234 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.082541 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.082545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.082547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.082698 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.083096 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.083103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.083433 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.083656 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.083979 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
I20260808 18:34:01.086138 3269 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:01.086387 3269 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:01.086892 3269 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:34:01.088140 2915 ts_manager.cc:194] Registered new tserver with Master: 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
I20260808 18:34:01.089035 2915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:49073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
I20260808 18:34:01.108080 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:01.222514 3274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:01.222750 3274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:01.222826 3274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:01.226997 3274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:01.227065 3274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:01.227186 3274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:01.232409 3274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:01.233475 3274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:01.234514 3274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:01.244849 3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.244869 3281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.244853 3284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:01.245148 3274 server_base.cc:1034] running on GCE node
I20260808 18:34:01.245584 3274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:01.246236 3274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:01.247443 3274 hybrid_clock.cc:648] HybridClock initialized: now 1786214041247414 us; error 49 us; skew 500 ppm
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:01.250254 3274 init.cc:377] Logged in from keytab as oryx/127.27.251.195@KRBTEST.COM (short username oryx)
I20260808 18:34:01.251180 3274 webserver.cc:460] Webserver started at http://127.27.251.195:35917/ using document root <none> and password file <none>
I20260808 18:34:01.251748 3274 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.251835 3274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.252089 3274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:01.254287 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "1bcaed098e384606ae0ad10645ed705a"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "bff9e6122eb275ded40e36f8cbfe1f9f"
server_key_iv: "cf28dc78d33ecb1d48e47be8a4ae7310"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.254987 3274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "1bcaed098e384606ae0ad10645ed705a"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "bff9e6122eb275ded40e36f8cbfe1f9f"
server_key_iv: "cf28dc78d33ecb1d48e47be8a4ae7310"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.258908 3274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 18:34:01.261574 3293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.263070 3274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:34:01.263196 3274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "1bcaed098e384606ae0ad10645ed705a"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "bff9e6122eb275ded40e36f8cbfe1f9f"
server_key_iv: "cf28dc78d33ecb1d48e47be8a4ae7310"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.263310 3274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.274588 3274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:01.277619 3274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:01.277822 3274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:01.278419 3274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:01.279269 3274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:01.279321 3274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.279389 3274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:01.279428 3274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.289181 3274 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:40191
I20260808 18:34:01.289217 3419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:40191 every 8 connection(s)
I20260808 18:34:01.290112 3274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:01.294956 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3274
I20260808 18:34:01.295095 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:01.295441 28655 external_mini_cluster.cc:1428] Setting key 95d3cc3804985ff4fe241cd2e1d435b5
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:01.304421 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.292060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:33003 (local address 127.27.251.254:36769)
0808 18:34:01.292203 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.292207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.293094 (+ 887us) server_negotiation.cc:407] Connection header received
0808 18:34:01.294763 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.294765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.294815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.294881 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.296404 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.297086 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.298243 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.298474 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.298588 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.300981 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.300989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.300992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.301015 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.302353 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.302925 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.302933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.302935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.303014 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.303407 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.303410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.303411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.303626 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.303714 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.304196 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.304301 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:01.305330 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.292421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:33003)
0808 18:34:01.292942 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.292986 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.294504 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.295121 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.295141 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.295639 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.296234 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.296252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.297239 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.297242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.298105 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.298114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.298630 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.298634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.298809 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.299335 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.299351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.300814 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.302480 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.302486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.302500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.302806 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.303119 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.303122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.303131 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.303274 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.303735 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.303741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.303978 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.304144 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.304417 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":131,"threads_started":1}
I20260808 18:34:01.306515 3420 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:01.306753 3420 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:01.307345 3420 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
I20260808 18:34:01.308463 2915 ts_manager.cc:194] Registered new tserver with Master: 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191)
I20260808 18:34:01.309064 2915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:33003
I20260808 18:34:01.309437 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.253.keytab.
Entry for principal oryx/127.27.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.253.keytab.
I20260808 18:34:01.339591 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:43435
--webserver_interface=127.27.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.253.keytab
--principal=oryx/127.27.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36769,127.27.251.253:43435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:01.455066 3426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:01.455365 3426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:01.455436 3426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:01.459487 3426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:01.459581 3426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:01.459617 3426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:01.459659 3426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:01.459692 3426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:01.465996 3426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36769,127.27.251.253:43435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.253.keytab
--principal=oryx/127.27.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
W20260808 18:34:01.466291 3426 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.251.254:36769,127.27.251.253:43435'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 18:34:01.467383 3426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:01.468773 3426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:01.478691 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.478672 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.478662 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:01.478672 3426 server_base.cc:1034] running on GCE node
I20260808 18:34:01.479480 3426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:01.480747 3426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:01.481963 3426 hybrid_clock.cc:648] HybridClock initialized: now 1786214041481924 us; error 48 us; skew 500 ppm
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:01.484895 3426 init.cc:377] Logged in from keytab as oryx/127.27.251.253@KRBTEST.COM (short username oryx)
I20260808 18:34:01.485904 3426 webserver.cc:460] Webserver started at http://127.27.251.253:36789/ using document root <none> and password file <none>
I20260808 18:34:01.486454 3426 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.486510 3426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.486697 3426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:01.488878 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/instance:
uuid: "a19f303970e24d1d904c1a0f43c13e37"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "afc193ef1241fcadcd100fe0239859b6"
server_key_iv: "1861c5b393ecab5c2ce81789cfd37455"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.489409 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal/instance:
uuid: "a19f303970e24d1d904c1a0f43c13e37"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "afc193ef1241fcadcd100fe0239859b6"
server_key_iv: "1861c5b393ecab5c2ce81789cfd37455"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.493448 3426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 18:34:01.495862 3443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.497172 3426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:34:01.497329 3426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "a19f303970e24d1d904c1a0f43c13e37"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "afc193ef1241fcadcd100fe0239859b6"
server_key_iv: "1861c5b393ecab5c2ce81789cfd37455"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.497444 3426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.509039 3426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:01.512167 3426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:01.512369 3426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:01.520090 3426 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.253:43435
I20260808 18:34:01.520392 3499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.253:43435 every 8 connection(s)
I20260808 18:34:01.521250 3426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
I20260808 18:34:01.524075 3500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:01.526675 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3426
I20260808 18:34:01.528962 3500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.253@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:01.541879 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.531317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.27.251.254:36769)
0808 18:34:01.531450 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.531455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.531925 (+ 470us) server_negotiation.cc:407] Connection header received
0808 18:34:01.532101 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.532103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.532148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.532235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.533089 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.533590 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.534656 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.534976 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.535108 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.538187 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.538195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.538198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.538221 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.539821 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.540387 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.540393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.540398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.540498 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.540808 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.540810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.540811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.541031 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.541171 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.541530 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.541654 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:34:01.542713 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.531258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36728)
0808 18:34:01.531798 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.531838 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.531985 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.532290 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.532296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.532770 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.532952 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.532962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.533729 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.533734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.534518 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.534529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.535328 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.535334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.535465 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.536049 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.536066 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.538031 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.539950 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.539957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.539981 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.540296 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.540618 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.540621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.540623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.540712 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.541148 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.541160 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.541310 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.541486 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.541918 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":103,"thread_start_us":307,"threads_started":1}
I20260808 18:34:01.542716 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.528383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.27.251.253:43435)
0808 18:34:01.528862 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.528880 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.528923 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:34:01.529764 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.529784 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.530182 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.530529 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.531408 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.532826 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.533797 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.534293 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.534409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.537636 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.537645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.537661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.537690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.539532 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.540028 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.540034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.540036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.540161 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.540465 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.540467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.540468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.541059 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.541254 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.541490 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.541808 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":150,"threads_started":1}
I20260808 18:34:01.544873 3500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.253@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
I20260808 18:34:01.552053 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.545508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.27.251.253:43435)
0808 18:34:01.545623 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.545632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.545644 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:34:01.545710 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.545712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.545744 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.545804 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.546156 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.546642 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.547327 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.547537 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.547608 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.549454 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.549457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.549459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.549478 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.550691 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.551058 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.551061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.551062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.551135 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.551364 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.551366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.551367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.551523 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.551596 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.551850 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.551936 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:34:01.552053 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.545439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58708)
0808 18:34:01.545565 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.545578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.545651 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.545836 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.545838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.545966 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.546060 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.546064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.546727 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.546730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.547233 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.547240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.547625 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.547627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.547701 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.548102 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.548113 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.549335 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.550791 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.550794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.550795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.550977 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.551213 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.551215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.551217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.551289 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.551602 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.551604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.551677 (+ 73us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.551791 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.551883 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 18:34:01.555572 3500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Bootstrap starting.
I20260808 18:34:01.557883 3500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:01.558574 3500 log.cc:824] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:01.560382 3500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: No bootstrap required, opened a new log
I20260808 18:34:01.563593 3500 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } }
I20260808 18:34:01.563791 3500 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:01.563833 3500 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19f303970e24d1d904c1a0f43c13e37, State: Initialized, Role: FOLLOWER
I20260808 18:34:01.564225 3500 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } }
I20260808 18:34:01.564699 3506 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } } }
I20260808 18:34:01.564832 3506 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:01.565389 3500 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:34:01.570103 3517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:34:01.570197 3517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:34:01.571888 3426 master_runner.cc:428] Detected that this master a19f303970e24d1d904c1a0f43c13e37 is joining an existing cluster
I20260808 18:34:01.571941 3426 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.251.253:43435
I20260808 18:34:01.582597 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.574134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.27.251.254:36769)
0808 18:34:01.574256 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.574259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.574415 (+ 156us) server_negotiation.cc:407] Connection header received
0808 18:34:01.574541 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.574545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.574604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.574680 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.575360 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.576222 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.577101 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.577333 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.577450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.578761 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.578766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.578769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.578797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.580695 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.581237 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.581241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.581243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.581333 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.581665 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.581669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.581671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.581898 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.582005 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.582305 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.582429 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:34:01.582623 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.573905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36732)
0808 18:34:01.574314 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.574329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.574437 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.574728 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.574730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.575042 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.575223 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.575233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.576337 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.576339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.576979 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.576988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.577462 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.577466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.577580 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.578112 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.578124 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.578608 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.580824 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.580829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.580833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.581128 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.581428 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.581433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.581436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.581544 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.582017 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.582020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.582118 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.582238 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.582435 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":20,"thread_start_us":104,"threads_started":1}
I20260808 18:34:01.584228 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.573975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58716)
0808 18:34:01.574429 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.574444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.574599 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.574866 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.574870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.575097 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.575223 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.575244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.577288 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.577291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.578175 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.578186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.579108 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.579112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.579223 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.579640 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.579655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.580068 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.582173 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.582176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.582178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.582480 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.583215 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.583218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.583219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.583306 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.583697 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.583700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.583781 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.583879 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.584022 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260808 18:34:01.584348 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.573995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.27.251.253:43435)
0808 18:34:01.574111 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.574116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.574536 (+ 420us) server_negotiation.cc:407] Connection header received
0808 18:34:01.574702 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.574705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.574754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.574837 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.576377 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.577150 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.578304 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.578625 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.578722 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.580215 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.580219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.580223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.580247 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.582053 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.582586 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.582589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.582590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.583089 (+ 499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.583393 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.583396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.583398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.583575 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.583640 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.583954 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.584097 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":49}
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.254@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
I20260808 18:34:01.599759 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.587160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.27.251.253:43435)
0808 18:34:01.587267 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.587271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.588912 (+ 1641us) server_negotiation.cc:407] Connection header received
0808 18:34:01.589070 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.589073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.589134 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.589216 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.589862 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.590637 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.591481 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.591810 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.591910 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.594106 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.594111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.594114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.594147 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.595750 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.596336 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.596339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.596341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.596429 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.598712 (+ 2283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.598715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.598717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.598940 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.599043 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.599483 (+ 440us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.599599 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:34:01.599900 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.587125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58730)
0808 18:34:01.588776 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.588832 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.588947 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.589254 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.589259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.589644 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.589748 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.589754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.590753 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.590756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.591371 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.591380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.591919 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.591922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.591998 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.592558 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.592578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.593970 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.595864 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.595874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.595881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.596221 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.598447 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.598451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.598455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.598577 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.599065 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.599075 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.599225 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.599391 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.599646 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"thread_start_us":93,"threads_started":1}
I20260808 18:34:01.601519 2915 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.251.253:43435: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37"
member_type: NON_VOTER
last_known_addr {
host: "127.27.251.253"
port: 43435
}
attrs {
promote: true
}
}
dest_uuid: "f224c5b1dd8e45a48424e6c437279130"
cas_config_opid_index: -1
I20260808 18:34:01.602276 2915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } }
I20260808 18:34:01.604429 3529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } }
I20260808 18:34:01.605202 3470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:01.606161 3529 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.251.253:43435
I20260808 18:34:01.606583 3426 master.cc:537] Master@127.27.251.253:43435 shutting down...
I20260808 18:34:01.606796 3470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 18:34:01.607203 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } }
I20260808 18:34:01.607302 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.27.251.252@KRBTEST.COM; defaulting to no policy
I20260808 18:34:01.608737 3529 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:34:01.610618 2901 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.252@KRBTEST.COM" created.
W20260808 18:34:01.611634 2901 consensus_peers.cc:487] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 -> Peer a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253:43435): Couldn't send request to peer a19f303970e24d1d904c1a0f43c13e37. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 18:34:01.612763 3426 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:01.613060 3426 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:01.613173 3426 tablet_replica.cc:330] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: stopping tablet replica
I20260808 18:34:01.629017 3426 master.cc:552] Master@127.27.251.253:43435 shutdown complete.
I20260808 18:34:01.629101 3426 master_runner.cc:305] Clearing existing system tablet
I20260808 18:34:01.629372 3426 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.629441 3426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.252.keytab.
Entry for principal oryx/127.27.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.252.keytab.
I20260808 18:34:01.631381 3426 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:01.632213 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:34995
--webserver_interface=127.27.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.252.keytab
--principal=oryx/127.27.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:34:01.632413 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.632685 3426 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.632865 3426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:01.632912 3426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "a19f303970e24d1d904c1a0f43c13e37"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "afc193ef1241fcadcd100fe0239859b6"
server_key_iv: "1861c5b393ecab5c2ce81789cfd37455"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.635293 3426 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:01.638377 3426 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 18:34:01.638489 3426 log.cc:1196] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 18:34:01.638778 3426 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Deleting consensus metadata
I20260808 18:34:01.638975 3426 master_runner.cc:315] Copying system tablet from 127.27.251.254:36769
I20260808 18:34:01.640406 3426 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: tablet copy: Beginning tablet copy session from remote peer at address 127.27.251.254:36769
I20260808 18:34:01.651068 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.641169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.27.251.254:36769)
0808 18:34:01.641315 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.641322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.641610 (+ 288us) server_negotiation.cc:407] Connection header received
0808 18:34:01.641842 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.641847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.641911 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.641979 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.642871 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.643849 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.645390 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.645811 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.645910 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.647478 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.647482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.647488 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.647514 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.649266 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.649782 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.649785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.649787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.649877 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.650173 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.650176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.650178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.650390 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.650487 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.650777 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.650895 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:34:01.651072 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.641063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36742)
0808 18:34:01.641498 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.641517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.641707 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.642211 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.642216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.642532 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.642705 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.642717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.644062 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.644068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.645226 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.645242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.646131 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.646137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.646265 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.646843 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.646859 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.647319 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.649398 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.649402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.649404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.649672 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.649976 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.649979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.649981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.650078 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.650490 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.650493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.650601 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.650712 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.650855 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":112,"threads_started":1}
I20260808 18:34:01.652012 2937 tablet_copy_service.cc:140] P f224c5b1dd8e45a48424e6c437279130: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a19f303970e24d1d904c1a0f43c13e37 ({username='oryx', principal='oryx/127.27.251.253@KRBTEST.COM'} at 127.0.0.1:36742)
I20260808 18:34:01.652218 2937 tablet_copy_service.cc:161] P f224c5b1dd8e45a48424e6c437279130: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a19f303970e24d1d904c1a0f43c13e37 at {username='oryx', principal='oryx/127.27.251.253@KRBTEST.COM'} at 127.0.0.1:36742: session id = a19f303970e24d1d904c1a0f43c13e37-00000000000000000000000000000000
I20260808 18:34:01.654219 2937 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 18:34:01.655907 3426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 18:34:01.658430 3426 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: tablet copy: Starting download of 0 data blocks...
I20260808 18:34:01.658636 3426 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: tablet copy: Starting download of 1 WAL segments...
I20260808 18:34:01.660465 3426 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 18:34:01.662386 2937 tablet_copy_service.cc:342] P f224c5b1dd8e45a48424e6c437279130: Request end of tablet copy session a19f303970e24d1d904c1a0f43c13e37-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.251.253@KRBTEST.COM'} at 127.0.0.1:36742
I20260808 18:34:01.662508 2937 tablet_copy_service.cc:433] P f224c5b1dd8e45a48424e6c437279130: ending tablet copy session a19f303970e24d1d904c1a0f43c13e37-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a19f303970e24d1d904c1a0f43c13e37
I20260808 18:34:01.670296 3426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:01.671999 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.672222 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.672438 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:01.673045 3426 server_base.cc:1034] running on GCE node
I20260808 18:34:01.673303 3426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:01.673619 3426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:01.674782 3426 hybrid_clock.cc:648] HybridClock initialized: now 1786214041674768 us; error 30 us; skew 500 ppm
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:01.676501 3426 init.cc:377] Logged in from keytab as oryx/127.27.251.253@KRBTEST.COM (short username oryx)
I20260808 18:34:01.677016 3426 webserver.cc:460] Webserver started at http://127.27.251.253:44675/ using document root <none> and password file <none>
I20260808 18:34:01.677222 3426 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.677268 3426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.678761 3426 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 18:34:01.679791 3557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.680091 3426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:01.680177 3426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "a19f303970e24d1d904c1a0f43c13e37"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "afc193ef1241fcadcd100fe0239859b6"
server_key_iv: "1861c5b393ecab5c2ce81789cfd37455"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.680259 3426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.701566 3426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:01.704536 3426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:01.710355 3426 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.253:43435
I20260808 18:34:01.710388 3613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.253:43435 every 8 connection(s)
I20260808 18:34:01.710999 3426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
I20260808 18:34:01.712867 3614 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:34:01.713917 3614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Bootstrap starting.
I20260808 18:34:01.724468 3614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:34:01.724864 3614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: Bootstrap complete.
I20260808 18:34:01.725454 3614 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } }
I20260808 18:34:01.725596 3614 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a19f303970e24d1d904c1a0f43c13e37, State: Initialized, Role: LEARNER
I20260808 18:34:01.725709 3614 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } }
I20260808 18:34:01.726054 3617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } }
I20260808 18:34:01.726200 3614 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:01.726241 3617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:34:01.729874 3629 catalog_manager.cc:1259] Loaded cluster ID: f6d51657d34645b794c56882bf5bec93
I20260808 18:34:01.729940 3629 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: loading cluster ID for follower catalog manager: success
I20260808 18:34:01.732007 3629 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: acquiring CA information for follower catalog manager: success
I20260808 18:34:01.733421 3629 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 18:34:01.770067 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:01.770346 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:01.770413 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:01.774369 3537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:01.774447 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:01.774484 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:01.774544 3537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:01.774588 3537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:01.779865 3537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.252.keytab
--principal=oryx/127.27.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:01.781117 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:01.782473 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:01.791376 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.791342 3637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:01.791342 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:01.791553 3537 server_base.cc:1034] running on GCE node
I20260808 18:34:01.792366 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:01.793514 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:01.794703 3537 hybrid_clock.cc:648] HybridClock initialized: now 1786214041794673 us; error 42 us; skew 500 ppm
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:01.797588 3537 init.cc:377] Logged in from keytab as oryx/127.27.251.252@KRBTEST.COM (short username oryx)
I20260808 18:34:01.798466 3537 webserver.cc:460] Webserver started at http://127.27.251.252:46575/ using document root <none> and password file <none>
I20260808 18:34:01.799033 3537 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.799125 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.799363 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:01.801541 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/instance:
uuid: "8031b29523b04cadb3f3337281c6b5f5"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "5324b2265e8540802be5d36c39e54013"
server_key_iv: "e22ed00a4e5555f8a4b1fa8945e645b1"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.802160 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal/instance:
uuid: "8031b29523b04cadb3f3337281c6b5f5"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "5324b2265e8540802be5d36c39e54013"
server_key_iv: "e22ed00a4e5555f8a4b1fa8945e645b1"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.805996 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:34:01.808588 3645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.809834 3537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:01.809991 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "8031b29523b04cadb3f3337281c6b5f5"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "5324b2265e8540802be5d36c39e54013"
server_key_iv: "e22ed00a4e5555f8a4b1fa8945e645b1"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.810111 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.825421 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:01.828840 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:01.829062 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:01.836404 3537 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.252:34995
I20260808 18:34:01.836732 3705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.252:34995 every 8 connection(s)
I20260808 18:34:01.837486 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
I20260808 18:34:01.840255 3706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:01.840356 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3537
I20260808 18:34:01.846097 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.252@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:01.856575 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.841988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.27.251.252:34995)
0808 18:34:01.842407 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.842426 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.842470 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:34:01.843602 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.843625 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.844070 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.844448 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.845165 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.846820 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.848006 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.848776 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.849353 (+ 577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.851915 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.851922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.851933 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.851968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.853792 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.854297 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.854302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.854308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.854418 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.854714 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.854718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.854720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.855093 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.855320 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.855569 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.855833 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260808 18:34:01.857820 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.848890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.27.251.254:36769)
0808 18:34:01.849020 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.849024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.849034 (+ 10us) server_negotiation.cc:407] Connection header received
0808 18:34:01.849070 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.849072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.849108 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.849205 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.850111 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.850878 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.851975 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.852209 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.852320 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.854467 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.854470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.854472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.854496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.855998 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.856451 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.856456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.856457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.856592 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.856904 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.856907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.856909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.857087 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.857208 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.857539 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.857653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:01.857846 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.847972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36752)
0808 18:34:01.848683 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.848738 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.848847 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.849255 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.849260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.849759 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.849952 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.849963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.851040 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.851044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.851827 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.851838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.852335 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.852339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.852497 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.852989 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.853000 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.854314 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.856107 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.856118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.856123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.856347 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.856690 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.856693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.856695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.856786 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.857222 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.857230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.857338 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.857478 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.857662 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":87,"threads_started":1}
I20260808 18:34:01.859018 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.252@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
I20260808 18:34:01.867303 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.859453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58740)
0808 18:34:01.859567 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.859581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.859661 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.860011 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.860012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.860137 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.860220 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.860225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.861194 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.861197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.862084 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.862094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.862665 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.862669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.862800 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.863165 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.863176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.864288 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.865819 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.865822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.865823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.866007 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.866299 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.866303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.866305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.866396 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.866786 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.866789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.866885 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.867006 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.867144 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 18:34:01.867358 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.859591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.27.251.253:43435)
0808 18:34:01.859819 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.859822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.859834 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:34:01.859871 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.859873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.859910 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.859985 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.860325 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.861055 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.862254 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.862528 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.862615 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.864415 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.864418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.864420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.864443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.865721 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.866094 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.866096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.866097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.866171 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.866496 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.866500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.866502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.866682 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.866794 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.867072 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.867195 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260808 18:34:01.868237 3706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:34:01.871367 3125 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.252@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
I20260808 18:34:01.875855 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.868713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34856)
0808 18:34:01.868814 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.868845 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.868920 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.869169 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.869172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.869316 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.869372 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.869376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.870434 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.870440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.870945 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.870952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.871431 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.871434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.871530 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.871997 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.872008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.873178 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.874595 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.874598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.874599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.874855 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.875107 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.875109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.875110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.875185 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.875492 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.875494 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.875584 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.875662 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.875749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260808 18:34:01.875934 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.868802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.27.251.252:34995)
0808 18:34:01.868910 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.868914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.868927 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:34:01.869019 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.869023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.869058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.869150 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.869481 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.869964 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.871062 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.871324 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.871429 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.873300 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.873303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.873305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.873326 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.874497 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.874947 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.874950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.874951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.875019 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.875264 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.875266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.875267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.875413 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.875486 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.875720 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.875805 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":49}
I20260808 18:34:01.878706 3706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Bootstrap starting.
I20260808 18:34:01.881071 3706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:01.881835 3706 log.cc:824] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:01.883606 3706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: No bootstrap required, opened a new log
I20260808 18:34:01.886714 3706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } }
I20260808 18:34:01.886916 3706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:01.886951 3706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8031b29523b04cadb3f3337281c6b5f5, State: Initialized, Role: FOLLOWER
I20260808 18:34:01.887379 3706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } }
I20260808 18:34:01.887897 3712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } } }
I20260808 18:34:01.888043 3712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:01.888639 3706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:34:01.893555 3724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:34:01.893640 3724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:34:01.897977 3537 master_runner.cc:428] Detected that this master 8031b29523b04cadb3f3337281c6b5f5 is joining an existing cluster
I20260808 18:34:01.898041 3537 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.251.252:34995
I20260808 18:34:01.909945 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2985
I20260808 18:34:01.912290 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.900694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.27.251.253:43435)
0808 18:34:01.900824 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.900828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.901224 (+ 396us) server_negotiation.cc:407] Connection header received
0808 18:34:01.901404 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.901409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.901464 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.901517 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.903465 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904296 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.905520 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.905815 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.905900 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.907110 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.907113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.907115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.907139 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.908793 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.909449 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.909453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.909458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.909559 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.909894 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.909897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.909899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.911361 (+ 1462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.911469 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.911921 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.912063 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:34:01.912765 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.900427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58756)
0808 18:34:01.901142 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.901157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.901286 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.901690 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.901693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.902835 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.903309 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.903319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904461 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.905376 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.905387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.905938 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.905942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.906056 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.906507 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.906522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.906978 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.909081 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.909084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.909086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.909345 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.909664 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.909667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.909670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.909764 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.911538 (+ 1774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.911545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.911687 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.912420 (+ 733us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.912629 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":125,"threads_started":1}
I20260808 18:34:01.923244 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:36307
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:36769
--webserver_port=45695
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:34:01.934209 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.901098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.27.251.252:34995)
0808 18:34:01.901209 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.901214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.901404 (+ 190us) server_negotiation.cc:407] Connection header received
0808 18:34:01.901551 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.901556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.901592 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.901638 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.902307 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.902968 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904205 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904566 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904649 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.929888 (+ 25239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.929893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.929895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.929920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.932160 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.932824 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.932827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.932829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.932906 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.933247 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.933251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.933252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.933525 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.933596 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.933902 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.934040 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:01.934209 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.900290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.27.251.254:36769)
0808 18:34:01.900426 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.900431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.900689 (+ 258us) server_negotiation.cc:407] Connection header received
0808 18:34:01.901067 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.901071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.901128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.901193 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.901950 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.902704 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904217 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904600 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904669 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.930322 (+ 25653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.930330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.930334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.930371 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.932161 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.932838 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.932842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.932844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.932924 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.933257 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.933261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.933262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.933469 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.933576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.933924 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.934040 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:01.934777 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.900806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34864)
0808 18:34:01.901331 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.901343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.901455 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.901793 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.901796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.902031 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.902180 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.902188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.903069 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.903075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.904066 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.904076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904816 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.905003 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.929159 (+ 24156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.929182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.929727 (+ 545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.932282 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.932285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.932287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.932715 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.933031 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.933033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.933035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.933148 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.933650 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.933663 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.933728 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.934407 (+ 679us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.934578 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":163,"spinlock_wait_cycles":10624,"thread_start_us":91,"threads_started":1}
I20260808 18:34:01.934793 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.900151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36768)
0808 18:34:01.900604 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.900621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.900849 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.901357 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.901362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.901696 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.901840 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.901849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.903096 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.903102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.904066 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.904076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.904816 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.904820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.905001 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.906041 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.906053 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.930127 (+ 24074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.932281 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.932285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.932287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.932715 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.933032 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.933034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.933036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.933157 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.933633 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.933638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.933730 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.934407 (+ 677us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.934630 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":165,"spinlock_wait_cycles":14976,"thread_start_us":107,"threads_started":1}
Aug 08 18:34:01 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214040, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.254@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
I20260808 18:34:01.946647 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.938605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.27.251.252:34995)
0808 18:34:01.938920 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.938923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.938932 (+ 9us) server_negotiation.cc:407] Connection header received
0808 18:34:01.938993 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.938995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.939033 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.939121 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.939518 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.940013 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.940901 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.941130 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.941241 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.943436 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.943440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.943442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.943474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.945017 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.945459 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.945462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.945464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.945556 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.945837 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.945842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.945844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.946018 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.946123 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.946383 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.946499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260808 18:34:01.946647 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.938568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34876)
0808 18:34:01.938735 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.938747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.938808 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.939140 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.939143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.939320 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.939401 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.939406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.940120 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.940125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.940783 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.940792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.941243 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.941247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.941335 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.941790 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.941804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.943303 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.945146 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.945150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.945152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.945348 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.945650 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.945653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.945655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.945744 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.946114 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.946117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.946221 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.946329 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.946478 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260808 18:34:01.947870 2915 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.251.252:34995: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5"
member_type: NON_VOTER
last_known_addr {
host: "127.27.251.252"
port: 34995
}
attrs {
promote: true
}
}
dest_uuid: "f224c5b1dd8e45a48424e6c437279130"
cas_config_opid_index: 5
I20260808 18:34:01.948499 2915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } }
I20260808 18:34:01.949792 3529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.951033 3529 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.251.252:34995
I20260808 18:34:01.951223 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.951344 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:01.952241 3537 master.cc:537] Master@127.27.251.252:34995 shutting down...
I20260808 18:34:01.952688 3678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:01.954342 3678 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 18:34:01.955659 3530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:34:01.957647 2901 consensus_peers.cc:487] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 -> Peer 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252:34995): Couldn't send request to peer 8031b29523b04cadb3f3337281c6b5f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 18:34:01.959672 3537 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:01.960031 3537 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:01.960211 3537 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: stopping tablet replica
I20260808 18:34:01.962044 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.951895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:58760)
0808 18:34:01.954036 (+ 2141us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.954053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.954150 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.954618 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.954621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.954640 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:01.954745 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.954755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.958793 (+ 4038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.958797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.960239 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.960253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.961463 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.961467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.961559 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.961613 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.961797 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.961887 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2031}
I20260808 18:34:01.962524 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.951990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.27.251.253:43435)
0808 18:34:01.952126 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.952131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.954246 (+ 2115us) server_negotiation.cc:407] Connection header received
0808 18:34:01.954311 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.954314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.954387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.954478 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:01.954897 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.957687 (+ 2790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.960403 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.961107 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.961236 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.961391 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.962292 (+ 901us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.962386 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:34:01.963074 3580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 LEARNER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 18:34:01.963579 3530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 18:34:01.965787 3580 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.967789 3537 master.cc:552] Master@127.27.251.252:34995 shutdown complete.
I20260808 18:34:01.967877 3537 master_runner.cc:305] Clearing existing system tablet
I20260808 18:34:01.967964 3617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f224c5b1dd8e45a48424e6c437279130. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.968092 3617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:34:01.968194 3617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: NON_VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: true } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.968286 3617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:34:01.968416 3537 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:01.968546 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:01.970273 3537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 18:34:01.970438 3530 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: attempting to promote NON_VOTER a19f303970e24d1d904c1a0f43c13e37 to VOTER
I20260808 18:34:01.970969 3530 consensus_queue.cc:237] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } }
I20260808 18:34:01.972532 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.972076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34882)
0808 18:34:01.972334 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.972412 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260808 18:34:01.972554 3580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 LEARNER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 18:34:01.973302 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.973078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34898)
0808 18:34:01.973191 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.973241 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:34995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 18:34:01.973455 3738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 18:34:01.973590 2901 consensus_peers.cc:487] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 -> Peer 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252:34995): Couldn't send request to peer 8031b29523b04cadb3f3337281c6b5f5. Status: Network error: Client connection negotiation failed: client connection to 127.27.251.252:34995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 18:34:01.973999 3739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:01.974195 3537 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:01.974313 3537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260808 18:34:01.974349 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "8031b29523b04cadb3f3337281c6b5f5"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "5324b2265e8540802be5d36c39e54013"
server_key_iv: "e22ed00a4e5555f8a4b1fa8945e645b1"
server_key_version: "encryptionkey@0"
I20260808 18:34:01.976388 3537 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:01.978423 3537 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 18:34:01.978542 3537 log.cc:1196] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 18:34:01.978827 3537 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Deleting consensus metadata
I20260808 18:34:01.979022 3537 master_runner.cc:315] Copying system tablet from 127.27.251.254:36769
I20260808 18:34:01.979923 3529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.981182 3537 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.251.254:36769
I20260808 18:34:01.981534 3738 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.981657 3738 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:01.981550 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.981779 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:01.981822 3580 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.983429 3736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:01.983534 3736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:01.992203 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.984189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.27.251.254:36769)
0808 18:34:01.984307 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:01.984312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:01.984579 (+ 267us) server_negotiation.cc:407] Connection header received
0808 18:34:01.984728 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:01.984731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:01.984783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:01.984867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:01.985415 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.985951 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.986778 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.987012 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.987118 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.988777 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:01.988781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:01.988783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:01.988810 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:01.990525 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.990951 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.990955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.990956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.991047 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:01.991338 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:01.991341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:01.991343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:01.991537 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:01.991646 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:01.991922 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 18:34:01.992035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:34:01.992223 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:01.984150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:36774)
0808 18:34:01.984472 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:01.984483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:01.984630 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:01.984886 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:01.984889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:01.985153 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:01.985298 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.985305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.986065 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.986070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.986649 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:01.986657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:01.987125 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:01.987128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:01.987635 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:01.988197 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:01.988209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:01.988642 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:01.990632 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.990637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.990639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.990846 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.991147 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:01.991150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:01.991152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:01.991241 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:01.991648 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:01.991652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:01.991754 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:34:01.991863 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 18:34:01.992027 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":80,"threads_started":1}
I20260808 18:34:01.992944 2937 tablet_copy_service.cc:140] P f224c5b1dd8e45a48424e6c437279130: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8031b29523b04cadb3f3337281c6b5f5 ({username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:36774)
I20260808 18:34:01.993057 2937 tablet_copy_service.cc:161] P f224c5b1dd8e45a48424e6c437279130: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8031b29523b04cadb3f3337281c6b5f5 at {username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:36774: session id = 8031b29523b04cadb3f3337281c6b5f5-00000000000000000000000000000000
I20260808 18:34:01.994122 2937 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 18:34:01.995329 3537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 18:34:01.997324 3537 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: tablet copy: Starting download of 0 data blocks...
I20260808 18:34:01.997537 3537 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: tablet copy: Starting download of 1 WAL segments...
I20260808 18:34:01.999119 3537 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 18:34:02.000697 2937 tablet_copy_service.cc:342] P f224c5b1dd8e45a48424e6c437279130: Request end of tablet copy session 8031b29523b04cadb3f3337281c6b5f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:36774
I20260808 18:34:02.000790 2937 tablet_copy_service.cc:433] P f224c5b1dd8e45a48424e6c437279130: ending tablet copy session 8031b29523b04cadb3f3337281c6b5f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8031b29523b04cadb3f3337281c6b5f5
I20260808 18:34:02.007287 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:02.009276 3749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.009611 3747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.009632 3746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:02.010645 3537 server_base.cc:1034] running on GCE node
I20260808 18:34:02.010821 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:02.011143 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:02.012295 3537 hybrid_clock.cc:648] HybridClock initialized: now 1786214042012302 us; error 42 us; skew 500 ppm
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:02.013827 3537 init.cc:377] Logged in from keytab as oryx/127.27.251.252@KRBTEST.COM (short username oryx)
I20260808 18:34:02.014302 3537 webserver.cc:460] Webserver started at http://127.27.251.252:40537/ using document root <none> and password file <none>
I20260808 18:34:02.014498 3537 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:02.014555 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:02.015640 3537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:02.016608 3756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.016875 3537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:02.016973 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "8031b29523b04cadb3f3337281c6b5f5"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "5324b2265e8540802be5d36c39e54013"
server_key_iv: "e22ed00a4e5555f8a4b1fa8945e645b1"
server_key_version: "encryptionkey@0"
I20260808 18:34:02.017061 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:02.030457 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:02.032626 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:02.037694 3537 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.252:34995
I20260808 18:34:02.038112 3811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.252:34995 every 8 connection(s)
I20260808 18:34:02.038431 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
I20260808 18:34:02.040081 3813 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:34:02.041069 3813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Bootstrap starting.
I20260808 18:34:02.050935 3813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:34:02.051259 3813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Bootstrap complete.
I20260808 18:34:02.051787 3813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } }
I20260808 18:34:02.051939 3813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8031b29523b04cadb3f3337281c6b5f5, State: Initialized, Role: LEARNER
I20260808 18:34:02.052062 3813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } }
I20260808 18:34:02.052446 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:02.052635 3813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:02.052661 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:34:02.056571 3829 catalog_manager.cc:1259] Loaded cluster ID: f6d51657d34645b794c56882bf5bec93
I20260808 18:34:02.056620 3829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: loading cluster ID for follower catalog manager: success
I20260808 18:34:02.058166 3829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: acquiring CA information for follower catalog manager: success
I20260808 18:34:02.059311 3829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:34:02.091703 3269 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
W20260808 18:34:02.092557 3734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:02.092893 3734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:02.092960 3734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:02.099488 3734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:02.099575 3734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:02.099665 3734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:02.107717 3734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:36307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=45695
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:02.109200 3734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:02.110641 3734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:02.123363 3835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.123375 3834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:02.123682 3734 server_base.cc:1034] running on GCE node
W20260808 18:34:02.123459 3837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:02.124374 3734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:02.125101 3734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:02.126302 3734 hybrid_clock.cc:648] HybridClock initialized: now 1786214042126286 us; error 30 us; skew 500 ppm
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:02.129418 3734 init.cc:377] Logged in from keytab as oryx/127.27.251.193@KRBTEST.COM (short username oryx)
I20260808 18:34:02.130271 3734 webserver.cc:460] Webserver started at http://127.27.251.193:45695/ using document root <none> and password file <none>
I20260808 18:34:02.130812 3734 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:02.130889 3734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:02.134999 3734 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:34:02.137349 3845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.138692 3734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:02.138792 3734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "bf727fe62da441c98127b57c40878529"
format_stamp: "Formatted at 2026-08-08 18:34:00 on dist-test-slave-fd7l"
server_key: "fc135280b6a10cc365bfcdfedb3b479c"
server_key_iv: "6832a2ce5da4c60c535a401e027abef7"
server_key_version: "encryptionkey@0"
I20260808 18:34:02.139371 3734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:02.175065 3734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:02.178516 3734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:02.178751 3734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:02.179402 3734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:02.180529 3734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:02.180601 3734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.180680 3734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:02.180727 3734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.191547 3734 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:36307
I20260808 18:34:02.191567 3965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:36307 every 8 connection(s)
I20260808 18:34:02.192775 3734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:02.202817 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3734
I20260808 18:34:02.203059 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3130
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
I20260808 18:34:02.214448 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.194817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:46291 (local address 127.27.251.254:36769)
0808 18:34:02.194953 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.194958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.196018 (+ 1060us) server_negotiation.cc:407] Connection header received
0808 18:34:02.197258 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.197261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.197323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.197406 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.199904 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.200696 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.201950 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.202322 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.202441 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.210155 (+ 7714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.210160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.210162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.210202 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.211964 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212478 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.212483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.212485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.212605 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212950 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.212953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.212955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.213190 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.213273 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.214133 (+ 860us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.214267 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:02.215576 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.195185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:46291)
0808 18:34:02.195844 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.195907 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.197024 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.197629 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.197640 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.198222 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.198815 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.198824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.200812 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.200816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.201794 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.201804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.202484 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.202488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.203590 (+ 1102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.204242 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.204264 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.209993 (+ 5729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.212097 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.212102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.212371 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.212706 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.212711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.212829 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.213343 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.213351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.213817 (+ 466us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.214883 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.215024 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":250,"spinlock_wait_cycles":44416,"thread_start_us":130,"threads_started":1}
I20260808 18:34:02.215660 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.196418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:50333 (local address 127.27.251.253:43435)
0808 18:34:02.196571 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.196576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.196662 (+ 86us) server_negotiation.cc:407] Connection header received
0808 18:34:02.197567 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.197571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.197631 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.197717 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.200630 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.201439 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.202459 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.202743 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.202804 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.211391 (+ 8587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.211398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.211403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.211441 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.213458 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.213974 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.213981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.213996 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.214097 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.214463 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.214469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.214473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.214732 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.214810 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.215291 (+ 481us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.215429 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:02.215921 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.195185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.27.251.193:35219)
0808 18:34:02.195980 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.196014 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.197024 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.197637 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.197640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.198209 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.198802 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.198816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.199904 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.199908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.201141 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.201151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.201697 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.201701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.201940 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.202688 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.202711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.208888 (+ 6177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.211329 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.211339 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.211360 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.211995 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.212612 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.212623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.212836 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.213343 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.213351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.213817 (+ 466us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.214051 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.214422 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":121,"spinlock_wait_cycles":75520,"thread_start_us":154,"threads_started":1}
I20260808 18:34:02.216385 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.194668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:35219 (local address 127.27.251.252:34995)
0808 18:34:02.194953 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.194958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.196082 (+ 1124us) server_negotiation.cc:407] Connection header received
0808 18:34:02.197358 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.197361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.197438 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.197531 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.198966 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.199725 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.201268 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.201525 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.201645 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.209143 (+ 7498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.209150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.209156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.209195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.211193 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212113 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.212117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.212118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.212225 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.212948 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.212951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.212953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.213197 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.213283 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.216063 (+ 2780us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.216218 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260808 18:34:02.216329 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.196296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.27.251.193:50333)
0808 18:34:02.196560 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.196574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.197039 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.197748 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.197750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.198222 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.198815 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.198824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.201579 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.201582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.202316 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.202328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.202975 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.202978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.203073 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.208186 (+ 5113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.208198 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.210267 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.213591 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.213596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.213601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.213860 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.214224 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.214230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.214236 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.214343 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.214971 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.214977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.215100 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.215855 (+ 755us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.216041 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":114,"thread_start_us":89,"threads_started":1}
I20260808 18:34:02.217396 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:33077
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:36769
--webserver_port=36709
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:34:02.217870 3966 heartbeater.cc:347] Connected to a master server at 127.27.251.252:34995
I20260808 18:34:02.218158 3966 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.218719 3966 heartbeater.cc:511] Master 127.27.251.252:34995 requested a full tablet report, sending...
I20260808 18:34:02.219053 3968 heartbeater.cc:347] Connected to a master server at 127.27.251.253:43435
I20260808 18:34:02.219166 3968 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.219342 3968 heartbeater.cc:511] Master 127.27.251.253:43435 requested a full tablet report, sending...
I20260808 18:34:02.219553 3967 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:02.219645 3967 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.219817 3967 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
I20260808 18:34:02.220605 2915 ts_manager.cc:194] Re-registered known tserver with Master: bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:02.220649 3574 ts_manager.cc:194] Registered new tserver with Master: bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:02.220659 3773 ts_manager.cc:194] Registered new tserver with Master: bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:02.221324 2915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:46291
I20260808 18:34:02.312180 3420 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
W20260808 18:34:02.339742 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:02.340088 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:02.340199 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:02.344566 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:02.344658 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:02.344758 3975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:02.350383 3975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:33077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=36709
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:02.351550 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:02.352684 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:02.363415 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.363431 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.363431 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:02.363833 3975 server_base.cc:1034] running on GCE node
I20260808 18:34:02.364182 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:02.364845 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:02.366047 3975 hybrid_clock.cc:648] HybridClock initialized: now 1786214042366016 us; error 45 us; skew 500 ppm
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:02.369166 3975 init.cc:377] Logged in from keytab as oryx/127.27.251.194@KRBTEST.COM (short username oryx)
I20260808 18:34:02.370241 3975 webserver.cc:460] Webserver started at http://127.27.251.194:36709/ using document root <none> and password file <none>
I20260808 18:34:02.370875 3975 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:02.370950 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:02.372605 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.368323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:34914)
0808 18:34:02.368461 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.368477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.368622 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.369170 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.369174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.369193 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.369303 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.369310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.370512 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.370516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.371327 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.371337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.372237 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.372239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.372303 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.372324 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.372374 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.372434 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 18:34:02.372684 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.368782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.27.251.252:34995)
0808 18:34:02.368915 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.368919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.368931 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:34:02.368981 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.368983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.369048 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.369152 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.369456 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.370244 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.371456 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.372117 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.372256 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.372410 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.372448 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.372560 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:02.373267 3783 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 18:34:02.373755 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f224c5b1dd8e45a48424e6c437279130. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:02.373922 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: LEARNER
I20260808 18:34:02.375432 3975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:02.377481 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.378739 3975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:02.378834 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "7b2064f1f9d04afe894fa263e6309409"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "44e20cf4a68352658767d26ec4a0fa9a"
server_key_iv: "9e44e2eed3478b575d5956cf7253cb2a"
server_key_version: "encryptionkey@0"
I20260808 18:34:02.379348 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:02.394155 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:02.397188 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:02.397346 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:02.397918 3975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:02.398764 3975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:02.398813 3975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.398854 3975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:02.398869 3975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.408236 3975 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:33077
I20260808 18:34:02.408259 4114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:33077 every 8 connection(s)
I20260808 18:34:02.409348 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:02.416838 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 3975
I20260808 18:34:02.417037 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3274
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:02.426496 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.411259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:40225 (local address 127.27.251.254:36769)
0808 18:34:02.411391 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.411396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.412650 (+ 1254us) server_negotiation.cc:407] Connection header received
0808 18:34:02.413919 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.413924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.413985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.414053 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.415617 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416434 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.418646 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.419047 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.419130 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.422031 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.422035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.422038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.422073 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.423705 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.424263 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.424267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.424269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.424362 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.425166 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.425170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.425172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.425382 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.425462 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.426175 (+ 713us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.426311 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:34:02.426957 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.411257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:53307 (local address 127.27.251.253:43435)
0808 18:34:02.411391 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.411396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.412724 (+ 1328us) server_negotiation.cc:407] Connection header received
0808 18:34:02.413973 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.413976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.414032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.414102 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.415624 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416434 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.417704 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.417970 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.418059 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.422078 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.422082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.422085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.422115 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.423975 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.424502 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.424536 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.424541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.424646 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.425115 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.425122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.425127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.425405 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.425511 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.426509 (+ 998us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.426703 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:34:02.428486 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.411764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:40225)
0808 18:34:02.412443 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.412528 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.413640 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.414252 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.414262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.414845 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.415448 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.415468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416610 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.416615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.418483 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.418498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.419286 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.419291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.419402 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.419871 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.419891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.421855 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.423832 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.423839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.423855 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.424130 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.424773 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.424779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.424784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.425002 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.425542 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.425549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.425880 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.426727 (+ 847us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.427312 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":547,"spinlock_wait_cycles":46592,"thread_start_us":162,"threads_started":1}
I20260808 18:34:02.428489 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.411764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.27.251.194:53307)
0808 18:34:02.412636 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.412650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.413640 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.414252 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.414262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.414847 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.415448 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.415468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416595 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.416598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.417574 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.417585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.418224 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.418228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.418420 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.419109 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.419135 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.421888 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.424115 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.424118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.424120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.424360 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.424756 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.424762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.424767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.424991 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.425560 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.425566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.425883 (+ 317us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.426085 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.427253 (+ 1168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":184,"spinlock_wait_cycles":78592,"thread_start_us":157,"threads_started":1}
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
I20260808 18:34:02.430230 4116 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:02.430474 4116 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.431103 4116 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
I20260808 18:34:02.431154 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:40191
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:36769
--webserver_port=35917
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:34:02.431668 4117 heartbeater.cc:347] Connected to a master server at 127.27.251.253:43435
I20260808 18:34:02.431769 4117 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.431936 4117 heartbeater.cc:511] Master 127.27.251.253:43435 requested a full tablet report, sending...
I20260808 18:34:02.432854 3574 ts_manager.cc:194] Registered new tserver with Master: 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
I20260808 18:34:02.433022 2915 ts_manager.cc:194] Re-registered known tserver with Master: 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
I20260808 18:34:02.433751 2915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:40225
I20260808 18:34:02.433831 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.413039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:54037 (local address 127.27.251.252:34995)
0808 18:34:02.413162 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.413166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.413271 (+ 105us) server_negotiation.cc:407] Connection header received
0808 18:34:02.413868 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.413871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.413931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.414054 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.415595 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416123 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.425783 (+ 9660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.426146 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.426227 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.429792 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.429798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.429801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.429832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.431690 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.432233 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.432239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.432244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.432336 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.432711 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.432717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.432722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.432986 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.433074 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.433466 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.433629 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:02.434367 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.412928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.27.251.194:54037)
0808 18:34:02.413177 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.413190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.413645 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.414253 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.414260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.414841 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.415449 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.415464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.416273 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.416276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.425644 (+ 9368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.425651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.426405 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.426450 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.426704 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.427212 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.427229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.429623 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.431828 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.431834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.431839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.432113 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.432457 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.432463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.432468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.432600 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.433164 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.433169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.433273 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.433983 (+ 710us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.434148 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":68,"thread_start_us":88,"threads_started":1}
I20260808 18:34:02.434998 4115 heartbeater.cc:347] Connected to a master server at 127.27.251.252:34995
I20260808 18:34:02.435088 4115 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.435227 4115 heartbeater.cc:511] Master 127.27.251.252:34995 requested a full tablet report, sending...
I20260808 18:34:02.435984 3773 ts_manager.cc:194] Registered new tserver with Master: 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
W20260808 18:34:02.557797 4122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:02.558100 4122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:02.558151 4122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:02.563030 4122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:02.563098 4122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:02.563163 4122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:02.569092 4122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:40191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=35917
--tserver_master_addrs=127.27.251.254:36769,127.27.251.253:43435,127.27.251.252:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:02.570132 4122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:02.571136 4122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:02.580953 4132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.580960 4130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:02.580960 4129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:02.581509 4122 server_base.cc:1034] running on GCE node
I20260808 18:34:02.581969 4122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:02.582655 4122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:02.583861 4122 hybrid_clock.cc:648] HybridClock initialized: now 1786214042583827 us; error 44 us; skew 500 ppm
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:02.586763 4122 init.cc:377] Logged in from keytab as oryx/127.27.251.195@KRBTEST.COM (short username oryx)
I20260808 18:34:02.587568 4122 webserver.cc:460] Webserver started at http://127.27.251.195:35917/ using document root <none> and password file <none>
I20260808 18:34:02.588085 4122 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:02.588157 4122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:02.591952 4122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260808 18:34:02.594018 4139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.595427 4122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:02.595592 4122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "1bcaed098e384606ae0ad10645ed705a"
format_stamp: "Formatted at 2026-08-08 18:34:01 on dist-test-slave-fd7l"
server_key: "bff9e6122eb275ded40e36f8cbfe1f9f"
server_key_iv: "cf28dc78d33ecb1d48e47be8a4ae7310"
server_key_version: "encryptionkey@0"
I20260808 18:34:02.596156 4122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:02.607465 4122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:02.610548 4122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:02.610750 4122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:02.611330 4122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:02.612196 4122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:02.612272 4122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.612341 4122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:02.612391 4122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:02.621575 4122 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:40191
I20260808 18:34:02.621699 4261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:40191 every 8 connection(s)
I20260808 18:34:02.622627 4122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:02.629737 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 4122
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.253@KRBTEST.COM
Aug 08 18:34:02 dist-test-slave-fd7l krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.252@KRBTEST.COM
I20260808 18:34:02.644977 2958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.624445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:55613 (local address 127.27.251.254:36769)
0808 18:34:02.624640 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.624643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.625809 (+ 1166us) server_negotiation.cc:407] Connection header received
0808 18:34:02.627185 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.627189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.627244 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.627326 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.628909 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629714 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631482 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.631823 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.636080 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.636084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.636088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.636119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.638334 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.642227 (+ 3893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.642233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.642238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.642341 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.643288 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.643292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.643296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.643547 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.643657 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.644653 (+ 996us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.644779 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 18:34:02.645541 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.632264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.27.251.253:43435)
0808 18:34:02.633908 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.633915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.633931 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:02.633973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.633978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.634030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.634128 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.636784 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.637551 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.638703 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.639055 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.639126 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.640954 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.640960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.640963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.640997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.642910 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.643489 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.643496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.643498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.643635 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.644053 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.644057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.644059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.644288 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.644464 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.644840 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.645367 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":85,"threads_started":1}
I20260808 18:34:02.645586 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.632698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.27.251.254:36769)
0808 18:34:02.632959 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.632963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.633909 (+ 946us) server_negotiation.cc:407] Connection header received
0808 18:34:02.635973 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.635978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.636051 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.636142 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.637344 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.638102 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.639230 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.639594 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.639665 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.641358 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.641364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.641369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.641398 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.643652 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.644159 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.644163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.644165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.644244 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.644606 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.644609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.644611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.644832 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.644927 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.645308 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.645422 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260808 18:34:02.646425 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.624937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:55613)
0808 18:34:02.625726 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.625740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.626893 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.627532 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.627542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.628155 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.628760 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.628767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629908 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.629912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.631343 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.631354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.631987 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.632234 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.633210 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.633227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.635911 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.641676 (+ 5765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.641683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.641700 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.642069 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.642953 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.642956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.642958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.643163 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.643721 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.643728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.644078 (+ 350us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.644313 (+ 235us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.645321 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":339,"spinlock_wait_cycles":409088,"thread_start_us":154,"threads_started":1}
I20260808 18:34:02.646929 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.633423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.27.251.252:34995)
0808 18:34:02.635064 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.635068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.635082 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:02.637037 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.637042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.637105 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.638337 (+ 1232us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.638444 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.639275 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.640589 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.640999 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.641494 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.642734 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.642738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.642741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.642769 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.644706 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.645195 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.645198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.645199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.645289 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.645595 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.645598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.645599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.645820 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.646526 (+ 706us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.646605 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.646736 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1423,"thread_start_us":79,"threads_started":1}
I20260808 18:34:02.648762 4263 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36769
I20260808 18:34:02.648895 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.626409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:57981 (local address 127.27.251.253:43435)
0808 18:34:02.626547 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.626550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.626563 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:34:02.627186 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.627189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.627244 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.627304 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.628884 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629416 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631492 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.631776 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631838 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.635731 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.635735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.635738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.635772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.639250 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647533 (+ 8283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.647539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.647542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.647631 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647957 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.647960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.647962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.648197 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.648271 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.648636 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.648742 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:34:02.649016 4263 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.649132 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.624873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:59329 (local address 127.27.251.252:34995)
0808 18:34:02.625000 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.625004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.625733 (+ 729us) server_negotiation.cc:407] Connection header received
0808 18:34:02.627224 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.627228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.627290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.627390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.628908 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629706 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.631686 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.632005 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.632096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.639388 (+ 7292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.639392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.639395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.639423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.642831 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647112 (+ 4281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.647119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.647124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.647211 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647797 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.647802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.647807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.648023 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.648110 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.648872 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.648985 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:34:02.649416 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.626070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.27.251.195:57981)
0808 18:34:02.626336 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.626350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.626910 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.627568 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.627571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.628140 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.628742 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.628757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629639 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.629645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.631349 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.631360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.632006 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.632009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.632241 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.633171 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.633201 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.635519 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.647169 (+ 11650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.647178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.647417 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.647754 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.647760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.647849 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.648348 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.648350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.648421 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.649073 (+ 652us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.649228 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":212,"spinlock_wait_cycles":13952,"thread_start_us":75,"threads_started":1}
I20260808 18:34:02.649586 4263 heartbeater.cc:511] Master 127.27.251.254:36769 requested a full tablet report, sending...
I20260808 18:34:02.649844 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.624937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.27.251.195:59329)
0808 18:34:02.625569 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.625628 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.626893 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.627532 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.627542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.628155 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:02.628758 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.628767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.629854 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.629857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.631574 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.631580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.632287 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.632291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.632418 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.634566 (+ 2148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:02.634580 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:02.636900 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:02.646777 (+ 9877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.646780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.646782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.647012 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.647324 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:02.647328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:02.647330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:02.647687 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:02.648316 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:02.648320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:02.648422 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.648550 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.648700 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":654,"spinlock_wait_cycles":10240,"thread_start_us":134,"threads_started":1}
I20260808 18:34:02.650061 4264 heartbeater.cc:347] Connected to a master server at 127.27.251.253:43435
I20260808 18:34:02.650149 4264 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.650331 4264 heartbeater.cc:511] Master 127.27.251.253:43435 requested a full tablet report, sending...
I20260808 18:34:02.650451 4262 heartbeater.cc:347] Connected to a master server at 127.27.251.252:34995
I20260808 18:34:02.650521 4262 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:02.650708 4262 heartbeater.cc:511] Master 127.27.251.252:34995 requested a full tablet report, sending...
I20260808 18:34:02.651129 2917 ts_manager.cc:194] Re-registered known tserver with Master: 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191)
I20260808 18:34:02.651115 3574 ts_manager.cc:194] Registered new tserver with Master: 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191)
I20260808 18:34:02.651317 3773 ts_manager.cc:194] Registered new tserver with Master: 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191)
I20260808 18:34:02.651788 2917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:55613
I20260808 18:34:02.649314 2915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:02.653606 2915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:02.656172 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: NON_VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: true } } }
I20260808 18:34:02.656309 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:02.659636 3530 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: attempting to promote NON_VOTER 8031b29523b04cadb3f3337281c6b5f5 to VOTER
I20260808 18:34:02.659992 3530 consensus_queue.cc:237] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:02.662562 3580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 18:34:02.662938 4280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 18:34:02.663098 3783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 LEARNER]: Refusing update from remote peer f224c5b1dd8e45a48424e6c437279130: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 18:34:02.663463 4280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 18:34:02.665086 4281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.665514 3580 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.666272 3783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.666764 4281 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.666873 4280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.666992 4280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:02.667097 4280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.667169 4280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:02.666875 4281 sys_catalog.cc:458] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:02.667721 4289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.667850 4289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:02.668146 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f224c5b1dd8e45a48424e6c437279130" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:02.668233 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:02.674963 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:33077 (local address 127.0.0.1:41692)
0808 18:34:02.669687 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.669703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.669790 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.670902 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.670904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.670920 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.671011 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.671015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.672149 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.672156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.673426 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.673437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.674557 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.674560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.674673 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.674700 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.674751 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.674814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":125,"threads_started":1}
I20260808 18:34:02.675700 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:36307 (local address 127.0.0.1:50040)
0808 18:34:02.669986 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.669999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.670083 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.672201 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.672204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.672224 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.672331 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.672337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.673675 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.673679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.674444 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.674453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.675328 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.675332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.675413 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.675435 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.675497 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.675571 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260808 18:34:02.675753 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.27.251.193:36307)
0808 18:34:02.671835 (+ 1863us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.671853 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.671869 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:02.671927 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.671930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.672059 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.672223 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.672427 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.673564 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.674581 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.675197 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.675331 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.675468 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.675523 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.675617 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":98,"threads_started":1}
I20260808 18:34:02.675928 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:40191 (local address 127.0.0.1:35526)
0808 18:34:02.669371 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.669388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.669475 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.671547 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.671550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.671568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.671655 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.671664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.673160 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.673164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.674247 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.674258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.675568 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.675572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.675662 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.675684 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.675736 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.675797 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":3}
I20260808 18:34:02.676421 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.27.251.194:33077)
0808 18:34:02.670380 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.670390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.670411 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:34:02.670480 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.670484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.670738 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.670926 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.671141 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.672019 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.673563 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.674425 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.675190 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.676106 (+ 916us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.676168 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.676270 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":121,"threads_started":1}
I20260808 18:34:02.677044 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.669397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.27.251.195:40191)
0808 18:34:02.671186 (+ 1789us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.671195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.671213 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:34:02.671273 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.671276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.671408 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.671559 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.671791 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.673013 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.674409 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.675433 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.676636 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.676772 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.676819 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.676904 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1698,"thread_start_us":145,"threads_started":1}
I20260808 18:34:02.678992 4043 tablet_service.cc:1467] Processing CreateTablet for tablet 3883249fcc724d3588b80ef6d060a73b (DEFAULT_TABLE table=test-table [id=c7e6d148f9ea49b089fdcb10c9fe8a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:02.678987 3896 tablet_service.cc:1467] Processing CreateTablet for tablet 3883249fcc724d3588b80ef6d060a73b (DEFAULT_TABLE table=test-table [id=c7e6d148f9ea49b089fdcb10c9fe8a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:02.680154 3896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3883249fcc724d3588b80ef6d060a73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:02.680152 4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3883249fcc724d3588b80ef6d060a73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:02.679767 4192 tablet_service.cc:1467] Processing CreateTablet for tablet 3883249fcc724d3588b80ef6d060a73b (DEFAULT_TABLE table=test-table [id=c7e6d148f9ea49b089fdcb10c9fe8a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:02.680787 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3883249fcc724d3588b80ef6d060a73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:02.688649 4299 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Bootstrap starting.
I20260808 18:34:02.689379 4301 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Bootstrap starting.
I20260808 18:34:02.691100 4299 tablet_bootstrap.cc:654] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:02.691587 4301 tablet_bootstrap.cc:654] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:02.692148 4299 log.cc:824] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:02.692368 4301 log.cc:824] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:02.692893 4300 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Bootstrap starting.
I20260808 18:34:02.694793 4301 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: No bootstrap required, opened a new log
I20260808 18:34:02.694793 4299 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: No bootstrap required, opened a new log
I20260808 18:34:02.694989 4301 ts_tablet_manager.cc:1397] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:34:02.694989 4299 ts_tablet_manager.cc:1397] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260808 18:34:02.694995 4300 tablet_bootstrap.cc:654] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:02.695688 4300 log.cc:824] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:02.697639 4300 tablet_bootstrap.cc:492] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: No bootstrap required, opened a new log
I20260808 18:34:02.697847 4300 ts_tablet_manager.cc:1397] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:34:02.699303 4301 raft_consensus.cc:348] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.699600 4301 raft_consensus.cc:374] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:02.699653 4301 raft_consensus.cc:729] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf727fe62da441c98127b57c40878529, State: Initialized, Role: FOLLOWER
I20260808 18:34:02.699793 4299 raft_consensus.cc:348] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.700055 4299 raft_consensus.cc:374] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:02.700112 4299 raft_consensus.cc:729] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2064f1f9d04afe894fa263e6309409, State: Initialized, Role: FOLLOWER
I20260808 18:34:02.700132 4301 consensus_queue.cc:260] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.700655 4299 consensus_queue.cc:260] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.701197 4301 ts_tablet_manager.cc:1428] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:34:02.701596 4299 ts_tablet_manager.cc:1428] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260808 18:34:02.701606 3967 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
I20260808 18:34:02.702785 4116 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
I20260808 18:34:02.703976 4300 raft_consensus.cc:348] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.704262 4300 raft_consensus.cc:374] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:02.704319 4300 raft_consensus.cc:729] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcaed098e384606ae0ad10645ed705a, State: Initialized, Role: FOLLOWER
I20260808 18:34:02.704860 4300 consensus_queue.cc:260] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.705931 4300 ts_tablet_manager.cc:1428] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260808 18:34:02.706295 4263 heartbeater.cc:503] Master 127.27.251.254:36769 was elected leader, sending a full tablet report...
I20260808 18:34:02.749872 4306 raft_consensus.cc:482] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:02.750118 4306 raft_consensus.cc:504] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.751253 4306 leader_election.cc:288] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191), bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:02.755610 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.751588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:56465 (local address 127.27.251.195:40191)
0808 18:34:02.751753 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.751759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.751775 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:02.751866 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.751869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.751924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.752035 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.752378 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.753248 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.754531 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.755044 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.755197 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.755252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.755407 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.755477 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:34:02.755632 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.751927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:34997 (local address 127.27.251.193:36307)
0808 18:34:02.752079 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.752083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.752097 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:02.752149 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.752151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.752195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.752253 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:02.752561 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.753279 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.754548 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.755041 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.755195 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.755252 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.755430 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.755504 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:34:02.755658 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.751477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:40191 (local address 127.27.251.194:56465)
0808 18:34:02.751629 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.751647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.751767 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.752105 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.752107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.752118 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.752260 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.752268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.753378 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.753381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.754407 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.754417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.755168 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.755171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.755270 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.755303 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.755390 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.755464 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":73}
I20260808 18:34:02.755664 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.751821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:36307 (local address 127.27.251.194:34997)
0808 18:34:02.751931 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:02.751943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:02.752008 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:02.752298 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:02.752303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:02.752325 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:02.752429 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.752434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.753378 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.753381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.754414 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:02.754426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.755186 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.755190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:02.755303 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.755322 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:34:02.755411 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:34:02.755482 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":87}
I20260808 18:34:02.756304 4213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3883249fcc724d3588b80ef6d060a73b" candidate_uuid: "7b2064f1f9d04afe894fa263e6309409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcaed098e384606ae0ad10645ed705a" is_pre_election: true
I20260808 18:34:02.756399 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3883249fcc724d3588b80ef6d060a73b" candidate_uuid: "7b2064f1f9d04afe894fa263e6309409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf727fe62da441c98127b57c40878529" is_pre_election: true
I20260808 18:34:02.756685 4213 raft_consensus.cc:2454] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2064f1f9d04afe894fa263e6309409 in term 0.
I20260808 18:34:02.756726 3917 raft_consensus.cc:2454] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b2064f1f9d04afe894fa263e6309409 in term 0.
I20260808 18:34:02.757179 3995 leader_election.cc:302] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b2064f1f9d04afe894fa263e6309409, bf727fe62da441c98127b57c40878529; no voters:
I20260808 18:34:02.757426 4306 raft_consensus.cc:2781] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:02.757526 4306 raft_consensus.cc:482] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:02.757587 4306 raft_consensus.cc:3037] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:02.758697 4306 raft_consensus.cc:504] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.759039 4306 leader_election.cc:288] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [CANDIDATE]: Term 1 election: Requested vote from peers 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191), bf727fe62da441c98127b57c40878529 (127.27.251.193:36307)
I20260808 18:34:02.759441 4213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3883249fcc724d3588b80ef6d060a73b" candidate_uuid: "7b2064f1f9d04afe894fa263e6309409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcaed098e384606ae0ad10645ed705a"
I20260808 18:34:02.759492 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3883249fcc724d3588b80ef6d060a73b" candidate_uuid: "7b2064f1f9d04afe894fa263e6309409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf727fe62da441c98127b57c40878529"
I20260808 18:34:02.759594 3917 raft_consensus.cc:3037] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:02.759578 4213 raft_consensus.cc:3037] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:02.760960 3917 raft_consensus.cc:2454] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2064f1f9d04afe894fa263e6309409 in term 1.
I20260808 18:34:02.761003 4213 raft_consensus.cc:2454] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b2064f1f9d04afe894fa263e6309409 in term 1.
I20260808 18:34:02.761274 3995 leader_election.cc:302] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcaed098e384606ae0ad10645ed705a, 7b2064f1f9d04afe894fa263e6309409; no voters:
I20260808 18:34:02.761469 4306 raft_consensus.cc:2781] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:02.761756 4306 raft_consensus.cc:686] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2064f1f9d04afe894fa263e6309409, State: Running, Role: LEADER
I20260808 18:34:02.762126 4306 consensus_queue.cc:237] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } }
I20260808 18:34:02.765899 2916 catalog_manager.cc:5495] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "7b2064f1f9d04afe894fa263e6309409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } health_report { overall_health: UNKNOWN } } }
I20260808 18:34:02.801398 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.797685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.27.251.194:33077)
0808 18:34:02.797822 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.797827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.797854 (+ 27us) server_negotiation.cc:407] Connection header received
0808 18:34:02.797948 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.797950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.798005 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.798119 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:02.798384 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.798908 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.799807 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.800037 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.800170 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.800532 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:02.800969 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:02.801107 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.801188 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.801248 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:34:02.807322 3917 raft_consensus.cc:1261] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Refusing update from remote peer 7b2064f1f9d04afe894fa263e6309409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:02.807343 4213 raft_consensus.cc:1261] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Refusing update from remote peer 7b2064f1f9d04afe894fa263e6309409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:02.808080 4308 consensus_queue.cc:1035] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:02.808262 4306 consensus_queue.cc:1035] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:02.820150 4313 mvcc.cc:205] Tried to move back new op lower bound from 7316332719331401728 to 7316332719156346880. Current Snapshot: MvccSnapshot[applied={T|T < 7316332719331401728}]
I20260808 18:34:02.827423 2916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:02.827615 2916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:02.832909 2916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f224c5b1dd8e45a48424e6c437279130: Sending DeleteTablet for 3 replicas of tablet 3883249fcc724d3588b80ef6d060a73b
I20260808 18:34:02.833671 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3883249fcc724d3588b80ef6d060a73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:02 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:35526
I20260808 18:34:02.833930 4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 3883249fcc724d3588b80ef6d060a73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:02 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:41692
I20260808 18:34:02.834383 3896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3883249fcc724d3588b80ef6d060a73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:02 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:50040
I20260808 18:34:02.835829 4327 tablet_replica.cc:330] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: stopping tablet replica
I20260808 18:34:02.836102 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 2884
I20260808 18:34:02.845752 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.834941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.27.251.253:43435)
0808 18:34:02.835118 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.835122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.835138 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:02.835179 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.835182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.835224 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.835327 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.836261 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.838148 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.839628 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.839983 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.840104 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.841917 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.841921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.841924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.841951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.843700 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.844216 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.844220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.844222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.844317 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.844636 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.844639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.844641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.844858 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.845055 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.845340 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.845461 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 18:34:02.848584 4327 raft_consensus.cc:2229] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:02.848755 4323 tablet_replica.cc:330] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: stopping tablet replica
I20260808 18:34:02.849128 4327 raft_consensus.cc:2258] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:02.849124 4323 raft_consensus.cc:2229] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:02.849622 4324 tablet_replica.cc:330] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: stopping tablet replica
I20260808 18:34:02.849990 4324 raft_consensus.cc:2229] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:02.850474 4324 raft_consensus.cc:2258] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:02.851084 4327 ts_tablet_manager.cc:1905] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:02.852105 4324 ts_tablet_manager.cc:1905] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:02.852845 4323 raft_consensus.cc:2258] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:02.855680 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.835492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.27.251.252:34995)
0808 18:34:02.835619 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.835637 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.845733 (+ 10096us) server_negotiation.cc:407] Connection header received
0808 18:34:02.845930 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.845936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.846000 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.846062 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.846970 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.847820 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.849189 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.849601 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.849696 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.851670 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.851673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.851676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.851707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.853599 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.854188 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.854191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.854193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.854282 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.854597 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.854601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.854602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.854821 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.854947 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.855362 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.855490 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:34:02.856245 4324 ts_tablet_manager.cc:1918] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:02.856395 4324 log.cc:1196] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/3883249fcc724d3588b80ef6d060a73b
I20260808 18:34:02.856477 4323 ts_tablet_manager.cc:1905] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:02.856956 4324 ts_tablet_manager.cc:1939] T 3883249fcc724d3588b80ef6d060a73b P 7b2064f1f9d04afe894fa263e6309409: Deleting consensus metadata
W20260808 18:34:02.858142 3994 connection.cc:441] server connection from 127.0.0.1:41692 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41692 (request call id 1) could send its response
I20260808 18:34:02.859277 4323 ts_tablet_manager.cc:1918] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:02.859382 4323 log.cc:1196] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/3883249fcc724d3588b80ef6d060a73b
I20260808 18:34:02.859735 4323 ts_tablet_manager.cc:1939] T 3883249fcc724d3588b80ef6d060a73b P 1bcaed098e384606ae0ad10645ed705a: Deleting consensus metadata
W20260808 18:34:02.860773 4141 connection.cc:441] server connection from 127.0.0.1:35526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35526 (request call id 1) could send its response
I20260808 18:34:02.861213 4327 ts_tablet_manager.cc:1918] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:02.861334 4327 log.cc:1196] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/3883249fcc724d3588b80ef6d060a73b
I20260808 18:34:02.861626 4327 ts_tablet_manager.cc:1939] T 3883249fcc724d3588b80ef6d060a73b P bf727fe62da441c98127b57c40878529: Deleting consensus metadata
W20260808 18:34:02.862766 3847 connection.cc:441] server connection from 127.0.0.1:50040 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50040 (request call id 1) could send its response
I20260808 18:34:02.867100 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.856010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.27.251.253:43435)
0808 18:34:02.856149 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.856153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.857095 (+ 942us) server_negotiation.cc:407] Connection header received
0808 18:34:02.857280 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.857284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.857338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.857390 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.859052 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.859775 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.860906 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.861244 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.861314 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.863562 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.863565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.863568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.863596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.865338 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.865818 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.865822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.865824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.865915 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.866216 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.866220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.866222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.866413 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.866555 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.866791 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.866908 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:02.867378 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:02.856431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.27.251.252:34995)
0808 18:34:02.857204 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:02.857210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:02.857471 (+ 261us) server_negotiation.cc:407] Connection header received
0808 18:34:02.857641 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:02.857646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:02.857704 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:02.857756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:02.858734 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.859556 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.860839 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:02.861172 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:02.861244 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:02.863911 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:02.863915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:02.863918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:02.863944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:02.865642 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.866061 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.866064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.866066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.866143 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:02.866439 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:02.866443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:02.866444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:02.866646 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:02.866742 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:02.867099 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 18:34:02.867213 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697}
I20260808 18:34:03.819433 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:03.818847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:33617)
0808 18:34:03.819180 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:03.819306 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
W20260808 18:34:03.819836 3967 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:03.821352 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:03.820858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:44219)
0808 18:34:03.821154 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:03.821251 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":69,"threads_started":1}
W20260808 18:34:03.821673 4116 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:03.822379 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:03.822005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:60059)
0808 18:34:03.822232 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:03.822302 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
W20260808 18:34:03.822702 4263 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.332626 4338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f224c5b1dd8e45a48424e6c437279130)
I20260808 18:34:04.332842 4338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:04.333115 4339 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f224c5b1dd8e45a48424e6c437279130)
I20260808 18:34:04.333289 4339 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:04.334033 4338 leader_election.cc:288] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253:43435)
I20260808 18:34:04.334260 4339 leader_election.cc:288] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252:34995)
I20260808 18:34:04.334637 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.334131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34908)
0808 18:34:04.334436 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.334521 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":64,"threads_started":1}
W20260808 18:34:04.334816 3759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.334939 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.334358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34924)
0808 18:34:04.334765 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.334846 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":80,"threads_started":1}
W20260808 18:34:04.335093 3560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.335954 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.335722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34940)
0808 18:34:04.335841 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.335899 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 18:34:04.336102 3560 leader_election.cc:334] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.336328 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.335657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34936)
0808 18:34:04.336156 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.336213 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20260808 18:34:04.336473 3759 leader_election.cc:334] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.339056 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.334344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43435 (local address 127.0.0.1:53666)
0808 18:34:04.335219 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.335233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:04.335356 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:04.335761 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:04.335763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:04.335796 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:04.335957 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:04.335968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.337027 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.337031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:04.337935 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:04.337945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.338611 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:04.338725 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:04.338772 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 18:34:04.338844 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:34:04.338923 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":653,"spinlock_wait_cycles":9856,"thread_start_us":141,"threads_started":1}
I20260808 18:34:04.339167 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.335083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:34995 (local address 127.0.0.1:41862)
0808 18:34:04.335503 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.335515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:04.335644 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:04.336016 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:04.336021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:04.336043 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:04.336228 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:04.336236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.337251 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.337254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:04.338094 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:04.338102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.338739 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:04.338804 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:04.338848 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 18:34:04.338926 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 18:34:04.338980 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":83,"thread_start_us":79,"threads_started":1}
I20260808 18:34:04.339178 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.334417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.27.251.253:43435)
0808 18:34:04.334782 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:04.334789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:04.335448 (+ 659us) server_negotiation.cc:407] Connection header received
0808 18:34:04.335499 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:04.335502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:04.335559 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:04.335640 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:04.336090 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.336914 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338070 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.338502 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338591 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:04.338629 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:04.338869 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 18:34:04.338952 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260808 18:34:04.339368 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.334849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.27.251.252:34995)
0808 18:34:04.335383 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:04.335387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:04.335588 (+ 201us) server_negotiation.cc:407] Connection header received
0808 18:34:04.335752 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:04.335755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:04.335811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:04.335863 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:04.336393 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.337140 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338208 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:04.338624 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:04.338766 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:04.338811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:04.338944 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 18:34:04.339007 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":464,"spinlock_wait_cycles":3840,"thread_start_us":116,"threads_started":1}
I20260808 18:34:04.339773 3580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8031b29523b04cadb3f3337281c6b5f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a19f303970e24d1d904c1a0f43c13e37" is_pre_election: true
I20260808 18:34:04.339926 3783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19f303970e24d1d904c1a0f43c13e37" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8031b29523b04cadb3f3337281c6b5f5" is_pre_election: true
I20260808 18:34:04.339982 3580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8031b29523b04cadb3f3337281c6b5f5 in term 1.
I20260808 18:34:04.340147 3783 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a19f303970e24d1d904c1a0f43c13e37 in term 1.
I20260808 18:34:04.340373 3757 leader_election.cc:302] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8031b29523b04cadb3f3337281c6b5f5, a19f303970e24d1d904c1a0f43c13e37; no voters: f224c5b1dd8e45a48424e6c437279130
I20260808 18:34:04.340482 3558 leader_election.cc:302] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8031b29523b04cadb3f3337281c6b5f5, a19f303970e24d1d904c1a0f43c13e37; no voters: f224c5b1dd8e45a48424e6c437279130
I20260808 18:34:04.340627 4338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 18:34:04.340691 4338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f224c5b1dd8e45a48424e6c437279130)
I20260808 18:34:04.340744 4338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:34:04.340778 4339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 18:34:04.340842 4339 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f224c5b1dd8e45a48424e6c437279130)
I20260808 18:34:04.340889 4339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:34:04.341832 4338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:04.341848 4339 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:04.342221 4338 leader_election.cc:288] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 election: Requested vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253:43435)
I20260808 18:34:04.342273 4339 leader_election.cc:288] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 election: Requested vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), 8031b29523b04cadb3f3337281c6b5f5 (127.27.251.252:34995)
I20260808 18:34:04.342545 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.342322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34942)
0808 18:34:04.342440 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.342490 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260808 18:34:04.342829 3580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8031b29523b04cadb3f3337281c6b5f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a19f303970e24d1d904c1a0f43c13e37"
I20260808 18:34:04.342908 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.342479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34954)
0808 18:34:04.342570 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.342802 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 18:34:04.343015 3580 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8031b29523b04cadb3f3337281c6b5f5 in current term 2: Already voted for candidate a19f303970e24d1d904c1a0f43c13e37 in this term.
I20260808 18:34:04.342960 3783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a19f303970e24d1d904c1a0f43c13e37" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8031b29523b04cadb3f3337281c6b5f5"
I20260808 18:34:04.343151 3783 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a19f303970e24d1d904c1a0f43c13e37 in current term 2: Already voted for candidate 8031b29523b04cadb3f3337281c6b5f5 in this term.
I20260808 18:34:04.343672 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.343432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34960)
0808 18:34:04.343585 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.343624 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 18:34:04.343720 4340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.343479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:34968)
0808 18:34:04.343637 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.343675 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260808 18:34:04.343778 3560 leader_election.cc:334] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
W20260808 18:34:04.343804 3759 leader_election.cc:334] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:04.343835 3560 leader_election.cc:302] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a19f303970e24d1d904c1a0f43c13e37; no voters: 8031b29523b04cadb3f3337281c6b5f5, f224c5b1dd8e45a48424e6c437279130
I20260808 18:34:04.343863 3759 leader_election.cc:302] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8031b29523b04cadb3f3337281c6b5f5; no voters: a19f303970e24d1d904c1a0f43c13e37, f224c5b1dd8e45a48424e6c437279130
I20260808 18:34:04.344003 4338 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 18:34:04.344084 4339 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 18:34:04.821110 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.820666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:45853)
0808 18:34:04.820952 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.821012 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260808 18:34:04.822715 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.822210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:58921)
0808 18:34:04.822549 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.822614 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
I20260808 18:34:04.823565 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:04.823227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:50765)
0808 18:34:04.823440 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:04.823491 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260808 18:34:05.822549 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.822011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:53111)
0808 18:34:05.822372 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.822450 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260808 18:34:05.823920 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.823460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:55813)
0808 18:34:05.823759 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.823824 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":69,"threads_started":1}
I20260808 18:34:05.824671 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.824221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:44285)
0808 18:34:05.824477 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.824574 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260808 18:34:05.892789 4355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:05.892935 4355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:05.893500 4355 leader_election.cc:288] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253:43435)
I20260808 18:34:05.894033 3580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8031b29523b04cadb3f3337281c6b5f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a19f303970e24d1d904c1a0f43c13e37" is_pre_election: true
I20260808 18:34:05.894182 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.893746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35200)
0808 18:34:05.894004 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.894079 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260808 18:34:05.894217 3580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8031b29523b04cadb3f3337281c6b5f5 in term 2.
I20260808 18:34:05.894562 3757 leader_election.cc:302] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8031b29523b04cadb3f3337281c6b5f5, a19f303970e24d1d904c1a0f43c13e37; no voters:
I20260808 18:34:05.894762 4355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 18:34:05.894845 4355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:05.894889 4355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:34:05.895107 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.894867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35202)
0808 18:34:05.894994 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.895050 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:34:05.895233 3759 leader_election.cc:334] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:05.896452 4355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:05.896838 4355 leader_election.cc:288] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 election: Requested vote from peers f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769), a19f303970e24d1d904c1a0f43c13e37 (127.27.251.253:43435)
I20260808 18:34:05.897173 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.896973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35208)
0808 18:34:05.897064 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.897110 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 18:34:05.897208 3580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8031b29523b04cadb3f3337281c6b5f5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a19f303970e24d1d904c1a0f43c13e37"
I20260808 18:34:05.897351 3580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:34:05.898319 3580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8031b29523b04cadb3f3337281c6b5f5 in term 3.
I20260808 18:34:05.898643 3757 leader_election.cc:302] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8031b29523b04cadb3f3337281c6b5f5, a19f303970e24d1d904c1a0f43c13e37; no voters:
I20260808 18:34:05.898801 4355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 18:34:05.898999 4355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [term 3 LEADER]: Becoming Leader. State: Replica: 8031b29523b04cadb3f3337281c6b5f5, State: Running, Role: LEADER
I20260808 18:34:05.899111 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.898817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35218)
0808 18:34:05.898936 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.898991 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 18:34:05.899210 3759 leader_election.cc:334] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
I20260808 18:34:05.899416 4355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } }
I20260808 18:34:05.900965 4358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8031b29523b04cadb3f3337281c6b5f5. Latest consensus state: current_term: 3 leader_uuid: "8031b29523b04cadb3f3337281c6b5f5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:05.901136 4358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:05.901466 4360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:05.903734 4360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:05.904086 4360 catalog_manager.cc:1259] Loaded cluster ID: f6d51657d34645b794c56882bf5bec93
I20260808 18:34:05.904130 4360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:05.904778 4360 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:05.905136 4360 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Loaded TSK: 0
I20260808 18:34:05.905792 4360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:34:05.929986 3772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:05.933115 3772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:05.936257 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.935953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35240)
0808 18:34:05.936112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.936176 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 18:34:05.936312 3580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [term 3 FOLLOWER]: Refusing update from remote peer 8031b29523b04cadb3f3337281c6b5f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 18:34:05.936794 4358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 18:34:05.937028 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.936783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.0.0.1:35252)
0808 18:34:05.936893 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.936945 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:34:05.937147 3759 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 -> Peer f224c5b1dd8e45a48424e6c437279130 (127.27.251.254:36769): Couldn't send request to peer f224c5b1dd8e45a48424e6c437279130. Status: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 18:34:05.938241 4362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8031b29523b04cadb3f3337281c6b5f5. Latest consensus state: current_term: 3 leader_uuid: "8031b29523b04cadb3f3337281c6b5f5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:05.938359 4362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:05.939478 4355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8031b29523b04cadb3f3337281c6b5f5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:05.939603 4355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:05.939981 4363 mvcc.cc:205] Tried to move back new op lower bound from 7316332732149964800 to 7316332732006584320. Current Snapshot: MvccSnapshot[applied={T|T < 7316332732149964800}]
I20260808 18:34:05.940486 4362 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8031b29523b04cadb3f3337281c6b5f5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f224c5b1dd8e45a48424e6c437279130" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36769 } } peers { permanent_uuid: "a19f303970e24d1d904c1a0f43c13e37" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43435 } attrs { promote: false } } peers { permanent_uuid: "8031b29523b04cadb3f3337281c6b5f5" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 34995 } attrs { promote: false } } }
I20260808 18:34:05.940699 4362 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19f303970e24d1d904c1a0f43c13e37 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:34:05.950038 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.945494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:36307 (local address 127.0.0.1:51740)
0808 18:34:05.945597 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.945613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:05.945723 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:05.946457 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:05.946459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:05.946475 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:05.946594 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.946602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.947601 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.947606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.948732 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.948743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.949665 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.949669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.949762 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.949789 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:34:05.949836 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 18:34:05.949904 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":19}
I20260808 18:34:05.950580 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.945709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:33077 (local address 127.0.0.1:47604)
0808 18:34:05.945937 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.945952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:05.946033 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:05.946894 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:05.946897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:05.946914 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:05.947035 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.947042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.948046 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.948050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.949322 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.949332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.950221 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.950226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.950323 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.950347 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:34:05.950393 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 18:34:05.950458 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":106,"thread_start_us":101,"threads_started":1}
I20260808 18:34:05.950645 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.945733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.27.251.193:36307)
0808 18:34:05.946191 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.946196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.946211 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:05.946265 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.946268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.946319 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.946449 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:05.946745 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.947494 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.948875 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.949538 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.950342 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.950402 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.950436 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.950507 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":109,"threads_started":1}
I20260808 18:34:05.950794 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.946034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:40191 (local address 127.0.0.1:43454)
0808 18:34:05.946254 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.946268 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:05.946351 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:05.947024 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:05.947027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:05.947045 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:05.947257 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.947265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.948247 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.948251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.949571 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.949580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.950365 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.950368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.950452 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.950470 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:34:05.950577 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:34:05.950648 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":23,"thread_start_us":102,"threads_started":1}
I20260808 18:34:05.951002 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.946379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.27.251.195:40191)
0808 18:34:05.946775 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.946780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.946794 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:05.946869 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.946874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.946922 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.947010 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:05.947385 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.948142 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.949689 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.950137 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.950257 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.950304 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.950607 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.950873 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":78,"threads_started":1}
I20260808 18:34:05.951130 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.946405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.27.251.194:33077)
0808 18:34:05.946653 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.946660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.946677 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:34:05.946733 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.946735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.946784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.946871 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:05.947178 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.947923 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.949459 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.950078 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.950799 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.950871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.950914 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.951004 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260808 18:34:05.951666 3896 tablet_service.cc:1467] Processing CreateTablet for tablet b6b44769370749ae83206f007abadfa2 (DEFAULT_TABLE table=test-table [id=5d35830ae7174ddd83635bbe40a82942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:05.951706 4192 tablet_service.cc:1467] Processing CreateTablet for tablet b6b44769370749ae83206f007abadfa2 (DEFAULT_TABLE table=test-table [id=5d35830ae7174ddd83635bbe40a82942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:05.951730 4043 tablet_service.cc:1467] Processing CreateTablet for tablet b6b44769370749ae83206f007abadfa2 (DEFAULT_TABLE table=test-table [id=5d35830ae7174ddd83635bbe40a82942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:05.951970 3896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b44769370749ae83206f007abadfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:05.951970 4192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b44769370749ae83206f007abadfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:05.951961 4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b44769370749ae83206f007abadfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:05.954885 4374 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Bootstrap starting.
I20260808 18:34:05.955005 4375 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Bootstrap starting.
I20260808 18:34:05.955919 4375 tablet_bootstrap.cc:654] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:05.955919 4374 tablet_bootstrap.cc:654] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:05.956643 4376 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529: Bootstrap starting.
I20260808 18:34:05.957563 4374 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: No bootstrap required, opened a new log
I20260808 18:34:05.957654 4376 tablet_bootstrap.cc:654] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:05.957711 4374 ts_tablet_manager.cc:1397] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:05.958113 4374 raft_consensus.cc:348] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.958240 4374 raft_consensus.cc:374] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:05.958283 4374 raft_consensus.cc:729] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2064f1f9d04afe894fa263e6309409, State: Initialized, Role: FOLLOWER
I20260808 18:34:05.958436 4374 consensus_queue.cc:260] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.958570 4375 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: No bootstrap required, opened a new log
I20260808 18:34:05.958662 4375 ts_tablet_manager.cc:1397] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 18:34:05.958741 4374 ts_tablet_manager.cc:1428] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:05.959061 4375 raft_consensus.cc:348] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.959172 4375 raft_consensus.cc:374] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:05.959244 4375 raft_consensus.cc:729] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcaed098e384606ae0ad10645ed705a, State: Initialized, Role: FOLLOWER
I20260808 18:34:05.959477 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.959183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:33601)
0808 18:34:05.959340 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.959398 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 18:34:05.959484 4375 consensus_queue.cc:260] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.959862 4375 ts_tablet_manager.cc:1428] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:05.960886 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.960642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:33807)
0808 18:34:05.960746 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.960814 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 18:34:05.961244 4376 tablet_bootstrap.cc:492] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529: No bootstrap required, opened a new log
I20260808 18:34:05.961331 4376 ts_tablet_manager.cc:1397] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:34:05.961912 4376 raft_consensus.cc:348] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.962044 4376 raft_consensus.cc:374] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:05.962077 4376 raft_consensus.cc:729] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf727fe62da441c98127b57c40878529, State: Initialized, Role: FOLLOWER
I20260808 18:34:05.962121 4115 heartbeater.cc:503] Master 127.27.251.252:34995 was elected leader, sending a full tablet report...
I20260808 18:34:05.962253 4376 consensus_queue.cc:260] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.962430 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.962158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:45683)
0808 18:34:05.962304 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.962354 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 18:34:05.963567 4384 raft_consensus.cc:482] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:05.963711 4384 raft_consensus.cc:504] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.963842 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.963096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:38815)
0808 18:34:05.963715 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.963776 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549}
I20260808 18:34:05.963932 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.963657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:52773)
0808 18:34:05.963811 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.963862 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 18:34:05.964219 4262 heartbeater.cc:503] Master 127.27.251.252:34995 was elected leader, sending a full tablet report...
I20260808 18:34:05.964792 4376 ts_tablet_manager.cc:1428] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:05.965657 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.965382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:56059)
0808 18:34:05.965524 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.965585 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 18:34:05.966205 3966 heartbeater.cc:503] Master 127.27.251.252:34995 was elected leader, sending a full tablet report...
I20260808 18:34:05.968109 4384 leader_election.cc:288] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf727fe62da441c98127b57c40878529 (127.27.251.193:36307), 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
I20260808 18:34:05.970237 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.966007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:56987 (local address 127.27.251.193:36307)
0808 18:34:05.966498 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.966507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.966522 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:05.966568 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.966571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.966619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.966689 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:05.967234 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.967981 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.969275 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.969665 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.969755 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.969793 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.970100 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.970158 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260808 18:34:05.970268 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.965834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:36307 (local address 127.27.251.195:56987)
0808 18:34:05.965922 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.965937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:05.966026 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:05.966708 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:05.966711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:05.966727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:05.966856 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.966864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.968093 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.968097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.969115 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.969125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.969789 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.969793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.969969 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.969994 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:34:05.970075 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 18:34:05.970138 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":43}
I20260808 18:34:05.970683 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b44769370749ae83206f007abadfa2" candidate_uuid: "1bcaed098e384606ae0ad10645ed705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf727fe62da441c98127b57c40878529" is_pre_election: true
I20260808 18:34:05.970826 3917 raft_consensus.cc:2454] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcaed098e384606ae0ad10645ed705a in term 0.
I20260808 18:34:05.971277 4142 leader_election.cc:302] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcaed098e384606ae0ad10645ed705a, bf727fe62da441c98127b57c40878529; no voters:
I20260808 18:34:05.971419 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.966193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:33077 (local address 127.27.251.195:53941)
0808 18:34:05.966470 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.966484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:05.966570 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:05.968268 (+ 1698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:05.968270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:05.968289 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:05.968391 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.968397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.969210 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.969217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.970202 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:05.970211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.971019 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.971022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:05.971120 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.971145 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:34:05.971208 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 18:34:05.971257 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":32,"thread_start_us":122,"threads_started":1}
I20260808 18:34:05.971429 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.966834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:53941 (local address 127.27.251.194:33077)
0808 18:34:05.966957 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.966962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.966973 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:34:05.967012 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.967014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.967054 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.967122 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:05.968586 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.969095 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.970324 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.970851 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.970979 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.971053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.971242 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.971325 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:34:05.971647 4384 raft_consensus.cc:2781] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:05.971717 4384 raft_consensus.cc:482] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:05.971757 4384 raft_consensus.cc:3037] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:05.971995 4065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b44769370749ae83206f007abadfa2" candidate_uuid: "1bcaed098e384606ae0ad10645ed705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2064f1f9d04afe894fa263e6309409" is_pre_election: true
I20260808 18:34:05.972224 4065 raft_consensus.cc:2454] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bcaed098e384606ae0ad10645ed705a in term 0.
I20260808 18:34:05.973011 4384 raft_consensus.cc:504] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.973313 4384 leader_election.cc:288] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [CANDIDATE]: Term 1 election: Requested vote from peers bf727fe62da441c98127b57c40878529 (127.27.251.193:36307), 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077)
I20260808 18:34:05.973682 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b44769370749ae83206f007abadfa2" candidate_uuid: "1bcaed098e384606ae0ad10645ed705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf727fe62da441c98127b57c40878529"
I20260808 18:34:05.973743 4065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b44769370749ae83206f007abadfa2" candidate_uuid: "1bcaed098e384606ae0ad10645ed705a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2064f1f9d04afe894fa263e6309409"
I20260808 18:34:05.973798 3917 raft_consensus.cc:3037] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:05.973887 4065 raft_consensus.cc:3037] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:05.974948 3917 raft_consensus.cc:2454] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcaed098e384606ae0ad10645ed705a in term 1.
I20260808 18:34:05.975101 4065 raft_consensus.cc:2454] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bcaed098e384606ae0ad10645ed705a in term 1.
I20260808 18:34:05.975220 4142 leader_election.cc:302] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bcaed098e384606ae0ad10645ed705a, bf727fe62da441c98127b57c40878529; no voters:
I20260808 18:34:05.975466 4384 raft_consensus.cc:2781] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:05.975591 4384 raft_consensus.cc:686] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 1 LEADER]: Becoming Leader. State: Replica: 1bcaed098e384606ae0ad10645ed705a, State: Running, Role: LEADER
I20260808 18:34:05.975838 4384 consensus_queue.cc:237] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } }
I20260808 18:34:05.976423 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.976191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:59681)
0808 18:34:05.976306 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.976364 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 18:34:05.978443 3773 catalog_manager.cc:5495] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bcaed098e384606ae0ad10645ed705a (127.27.251.195). New cstate: current_term: 1 leader_uuid: "1bcaed098e384606ae0ad10645ed705a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bcaed098e384606ae0ad10645ed705a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40191 } health_report { overall_health: HEALTHY } } }
I20260808 18:34:05.990522 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.987263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.27.251.195:40191)
0808 18:34:05.987359 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:05.987362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:05.987394 (+ 32us) server_negotiation.cc:407] Connection header received
0808 18:34:05.987543 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:05.987546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:05.987596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:05.987680 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:05.987881 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.988406 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.989254 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:05.989460 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:05.989535 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:05.989766 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:05.990182 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:05.990265 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:05.990323 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 18:34:05.990366 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 18:34:05.993198 3917 raft_consensus.cc:1261] T b6b44769370749ae83206f007abadfa2 P bf727fe62da441c98127b57c40878529 [term 1 FOLLOWER]: Refusing update from remote peer 1bcaed098e384606ae0ad10645ed705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:05.993383 4065 raft_consensus.cc:1261] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Refusing update from remote peer 1bcaed098e384606ae0ad10645ed705a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:05.993794 4384 consensus_queue.cc:1035] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf727fe62da441c98127b57c40878529" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:05.993966 4382 consensus_queue.cc:1035] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b2064f1f9d04afe894fa263e6309409" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:05.995622 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.995309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:52483)
0808 18:34:05.995468 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.995537 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 18:34:05.996294 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.995995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:49061)
0808 18:34:05.996163 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:05.996222 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 18:34:05.999225 4390 mvcc.cc:205] Tried to move back new op lower bound from 7316332732382793728 to 7316332732320100352. Current Snapshot: MvccSnapshot[applied={T|T < 7316332732382793728}]
I20260808 18:34:06.000262 4353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:05.999976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.194:33049)
0808 18:34:06.000123 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:06.000189 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 18:34:06.000592 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:06.000180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:59825)
0808 18:34:06.000437 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:06.000496 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260808 18:34:06.003348 4352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:06.003033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.193:42851)
0808 18:34:06.003221 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:06.003274 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260808 18:34:06.005959 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:06.005704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36769 (local address 127.27.251.195:34129)
0808 18:34:06.005848 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:06.005896 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 18:34:06.008278 3773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:06.008436 3773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:06.012389 3773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8031b29523b04cadb3f3337281c6b5f5: Sending DeleteTablet for 3 replicas of tablet b6b44769370749ae83206f007abadfa2
I20260808 18:34:06.013151 3896 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b44769370749ae83206f007abadfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:06 UTC) from {username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:51740
I20260808 18:34:06.013291 4043 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b44769370749ae83206f007abadfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:06 UTC) from {username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:47604
I20260808 18:34:06.013319 4192 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b44769370749ae83206f007abadfa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:06 UTC) from {username='oryx', principal='oryx/127.27.251.252@KRBTEST.COM'} at 127.0.0.1:43454
I20260808 18:34:06.013774 4401 tablet_replica.cc:330] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: stopping tablet replica
I20260808 18:34:06.013927 4401 raft_consensus.cc:2229] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:06.014082 4401 raft_consensus.cc:2258] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:06.014276 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3734
I20260808 18:34:06.015307 4402 tablet_replica.cc:330] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: stopping tablet replica
I20260808 18:34:06.015460 4402 raft_consensus.cc:2229] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:06.015770 4402 raft_consensus.cc:2258] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:06.017040 4401 ts_tablet_manager.cc:1905] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:06.017406 4402 ts_tablet_manager.cc:1905] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:06.019590 4401 ts_tablet_manager.cc:1918] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:06.019687 4401 log.cc:1196] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/b6b44769370749ae83206f007abadfa2
I20260808 18:34:06.019980 4401 ts_tablet_manager.cc:1939] T b6b44769370749ae83206f007abadfa2 P 7b2064f1f9d04afe894fa263e6309409: Deleting consensus metadata
I20260808 18:34:06.020686 3759 catalog_manager.cc:4841] TS 7b2064f1f9d04afe894fa263e6309409 (127.27.251.194:33077): tablet b6b44769370749ae83206f007abadfa2 (table test-table [id=5d35830ae7174ddd83635bbe40a82942]) successfully deleted
I20260808 18:34:06.020800 4402 ts_tablet_manager.cc:1918] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:06.020882 4402 log.cc:1196] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/b6b44769370749ae83206f007abadfa2
I20260808 18:34:06.021232 4402 ts_tablet_manager.cc:1939] T b6b44769370749ae83206f007abadfa2 P 1bcaed098e384606ae0ad10645ed705a: Deleting consensus metadata
I20260808 18:34:06.021941 3759 catalog_manager.cc:4841] TS 1bcaed098e384606ae0ad10645ed705a (127.27.251.195:40191): tablet b6b44769370749ae83206f007abadfa2 (table test-table [id=5d35830ae7174ddd83635bbe40a82942]) successfully deleted
W20260808 18:34:06.025532 4371 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:06.025219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:36307 (local address 127.0.0.1:51754)
0808 18:34:06.025360 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:06.025420 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:36307: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 18:34:06.025610 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3975
W20260808 18:34:06.029320 3759 catalog_manager.cc:4584] TS bf727fe62da441c98127b57c40878529 (127.27.251.193:36307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6b44769370749ae83206f007abadfa2: Network error: Client connection negotiation failed: client connection to 127.27.251.193:36307: connect: Connection reset by peer (error 104)
I20260808 18:34:06.036453 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 4122
I20260808 18:34:06.045711 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3426
I20260808 18:34:06.054705 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 3537
2026-08-08T18:34:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6979 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:34:06 dist-test-slave-fd7l krb5kdc[4405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:34:06 dist-test-slave-fd7l krb5kdc[4405](info): set up 2 sockets
Aug 08 18:34:06 dist-test-slave-fd7l krb5kdc[4405](info): commencing operation
krb5kdc: starting...
W20260808 18:34:07.154217 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab.
Entry for principal oryx/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab.
I20260808 18:34:07.284298 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:39133
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab
--principal=oryx/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:07.399650 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:07.400008 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:07.400092 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:07.404240 4418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:07.404363 4418 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 18:34:07.404397 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:07.404439 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:07.404497 4418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:07.404551 4418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:07.410310 4418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:39133
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.254.keytab
--principal=oryx/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:07.411476 4418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:07.412617 4418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:07.421495 4426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:07.421499 4424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:07.421518 4423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:07.421725 4418 server_base.cc:1034] running on GCE node
I20260808 18:34:07.422461 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:07.423660 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:07.424839 4418 hybrid_clock.cc:648] HybridClock initialized: now 1786214047424802 us; error 48 us; skew 500 ppm
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:07.427748 4418 init.cc:377] Logged in from keytab as oryx/127.27.251.254@KRBTEST.COM (short username oryx)
I20260808 18:34:07.428802 4418 webserver.cc:460] Webserver started at http://127.27.251.254:32851/ using document root <none> and password file <none>
I20260808 18:34:07.429425 4418 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:07.429490 4418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:07.429729 4418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:07.432027 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "65a50b8d1e9649e6bbf7140b9968e249"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "e7454d35e989f50c09fd0c2ed5bb51a2"
server_key_iv: "7615315026c98429ee19bbe79f1028f3"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.432690 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "65a50b8d1e9649e6bbf7140b9968e249"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "e7454d35e989f50c09fd0c2ed5bb51a2"
server_key_iv: "7615315026c98429ee19bbe79f1028f3"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.436618 4418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:34:07.439137 4434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:07.440385 4418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:07.440548 4418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "65a50b8d1e9649e6bbf7140b9968e249"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "e7454d35e989f50c09fd0c2ed5bb51a2"
server_key_iv: "7615315026c98429ee19bbe79f1028f3"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.440662 4418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:07.467456 4418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:07.471618 4418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:07.471870 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:07.481891 4418 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:39133
I20260808 18:34:07.481900 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:39133 every 8 connection(s)
I20260808 18:34:07.482942 4418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:07.483358 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 4418
I20260808 18:34:07.483505 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:07.483812 28655 external_mini_cluster.cc:1428] Setting key cd6f671fc3a3df2623d72604ff917b88
I20260808 18:34:07.486292 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:07.491990 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Bootstrap starting.
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:07.494378 4503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:07.495489 4503 log.cc:824] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:07.497982 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: No bootstrap required, opened a new log
I20260808 18:34:07.499408 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:07.485313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.27.251.254:39133)
0808 18:34:07.485795 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:07.485811 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:07.485849 (+ 38us) server_negotiation.cc:407] Connection header received
0808 18:34:07.486619 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:07.486640 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:07.486988 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:07.487355 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:07.488204 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.489295 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.490187 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.490672 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.490840 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:07.493512 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:07.493517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:07.493537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:07.493565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:07.496209 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.496737 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.496744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.496750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.496906 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.497263 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.497266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.497267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.497785 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:07.497985 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:07.498347 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 18:34:07.498652 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":145,"threads_started":1}
I20260808 18:34:07.501735 4503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } }
I20260808 18:34:07.502032 4503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:07.502116 4503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a50b8d1e9649e6bbf7140b9968e249, State: Initialized, Role: FOLLOWER
I20260808 18:34:07.502589 4503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } }
I20260808 18:34:07.502753 4503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:07.502821 4503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:07.502920 4503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:07.503882 4503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } }
I20260808 18:34:07.504230 4503 leader_election.cc:302] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a50b8d1e9649e6bbf7140b9968e249; no voters:
I20260808 18:34:07.504590 4503 leader_election.cc:288] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:07.504729 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:07.505014 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [term 1 LEADER]: Becoming Leader. State: Replica: 65a50b8d1e9649e6bbf7140b9968e249, State: Running, Role: LEADER
I20260808 18:34:07.505364 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } }
I20260808 18:34:07.505851 4503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:07.507017 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a50b8d1e9649e6bbf7140b9968e249. Latest consensus state: current_term: 1 leader_uuid: "65a50b8d1e9649e6bbf7140b9968e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } } }
I20260808 18:34:07.507153 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:07.507525 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a50b8d1e9649e6bbf7140b9968e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a50b8d1e9649e6bbf7140b9968e249" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39133 } } }
I20260808 18:34:07.507622 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:07.507598 4517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:07.510609 4517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:07.516086 4517 catalog_manager.cc:1347] Generated new cluster ID: a6c93100ac664a84a43ec8c5565860c9
I20260808 18:34:07.516172 4517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:07.530596 4517 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:07.531550 4517 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:07.540658 4517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Generated new TSK 0
I20260808 18:34:07.541396 4517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
I20260808 18:34:07.579282 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:39133
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:07.697613 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:07.698029 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:07.698112 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:07.702344 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:07.702414 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:07.702456 4533 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:34:07.702522 4533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:07.708250 4533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:39133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:07.709502 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:07.710625 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:07.721362 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:07.721369 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:07.721403 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:07.721783 4533 server_base.cc:1034] running on GCE node
I20260808 18:34:07.722272 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:07.723006 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:07.724195 4533 hybrid_clock.cc:648] HybridClock initialized: now 1786214047724155 us; error 47 us; skew 500 ppm
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:07.727104 4533 init.cc:377] Logged in from keytab as oryx/127.27.251.193@KRBTEST.COM (short username oryx)
I20260808 18:34:07.728011 4533 webserver.cc:460] Webserver started at http://127.27.251.193:42295/ using document root <none> and password file <none>
I20260808 18:34:07.728663 4533 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:07.728760 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:07.729017 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:07.731140 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "f14be40334fc4070a244ad995b5d88f0"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "9819aa803cdbcbb8b71a3b4d0c36ae2c"
server_key_iv: "eb010655ecd82f5dd14be0e285a577b9"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.731748 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "f14be40334fc4070a244ad995b5d88f0"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "9819aa803cdbcbb8b71a3b4d0c36ae2c"
server_key_iv: "eb010655ecd82f5dd14be0e285a577b9"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.735606 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:34:07.738324 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:07.739576 4533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:07.739722 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "f14be40334fc4070a244ad995b5d88f0"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "9819aa803cdbcbb8b71a3b4d0c36ae2c"
server_key_iv: "eb010655ecd82f5dd14be0e285a577b9"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.739841 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:07.760143 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:07.763482 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:07.763715 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:07.765412 4533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:07.765496 4533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:07.765568 4533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:07.765609 4533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:07.776923 4533 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:45123
I20260808 18:34:07.778115 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:07.779012 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:45123 every 8 connection(s)
I20260808 18:34:07.781389 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:07.766221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:34317 (local address 127.27.251.254:39133)
0808 18:34:07.766382 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:07.766386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:07.767454 (+ 1068us) server_negotiation.cc:407] Connection header received
0808 18:34:07.768827 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:07.768831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:07.768896 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:07.768995 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:07.770705 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.771357 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.772598 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.772935 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.773071 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:07.777116 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:07.777120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:07.777123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:07.777152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:07.778757 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.779418 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.779522 (+ 104us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.779523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.779632 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.780092 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.780097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.780099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.780321 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:07.780467 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:07.781041 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 18:34:07.781191 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:34:07.782285 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:07.766691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.193:34317)
0808 18:34:07.767251 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:07.767316 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:07.768564 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:07.769241 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:07.769254 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:07.769752 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:07.770521 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:07.770548 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.771502 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.771506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:07.772420 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:07.772430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.773242 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.773247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:07.773503 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:07.774891 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:07.774917 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:07.776917 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:07.778906 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:07.778913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:07.778942 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:07.779295 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:07.779783 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:07.779786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:07.779788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:07.779966 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:07.780468 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:07.780477 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:07.780763 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 18:34:07.780975 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 18:34:07.781321 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
I20260808 18:34:07.783748 4674 heartbeater.cc:347] Connected to a master server at 127.27.251.254:39133
I20260808 18:34:07.783895 4674 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:07.784467 4674 heartbeater.cc:511] Master 127.27.251.254:39133 requested a full tablet report, sending...
I20260808 18:34:07.786389 4450 ts_manager.cc:194] Registered new tserver with Master: f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123)
I20260808 18:34:07.787251 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 4533
I20260808 18:34:07.787421 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:07.787635 4450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:34317
I20260808 18:34:07.787850 28655 external_mini_cluster.cc:1428] Setting key b23380aa16f1e1929d301167261c8406
I20260808 18:34:07.798180 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:07.786235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:53921 (local address 127.27.251.254:39133)
0808 18:34:07.786373 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:07.786378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:07.788432 (+ 2054us) server_negotiation.cc:407] Connection header received
0808 18:34:07.790141 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:07.790147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:07.790208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:07.790304 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:07.790796 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.791652 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.792959 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.793183 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.793274 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:07.794666 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:07.794668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:07.794671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:07.794693 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:07.796220 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.796830 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.796832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.796833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.796910 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:07.797243 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:07.797245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:07.797246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:07.797416 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:07.797489 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:07.797822 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 18:34:07.798001 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:34:07.798190 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:07.786112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.193:53921)
0808 18:34:07.788356 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:07.788374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:07.788474 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:07.790330 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:07.790333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:07.790574 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:07.790675 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:07.790681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.791826 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.791833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:07.792807 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:07.792823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:07.793308 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:07.793314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:07.793441 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:07.793980 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:07.794008 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:07.794522 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:07.796341 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:07.796347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:07.796360 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:07.796725 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:07.797019 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:07.797023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:07.797025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:07.797139 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:07.797516 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:07.797519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:07.797640 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 18:34:07.797753 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 18:34:07.797913 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
I20260808 18:34:07.819885 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:39133
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:07.936307 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:07.936627 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:07.936678 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:07.940722 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:07.940776 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:07.940850 4686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:34:07.940930 4686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:07.946326 4686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:39133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:07.947376 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:07.948406 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:07.958638 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:07.958678 4686 server_base.cc:1034] running on GCE node
W20260808 18:34:07.958637 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:07.958637 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:07.959345 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:07.960024 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:07.961231 4686 hybrid_clock.cc:648] HybridClock initialized: now 1786214047961214 us; error 32 us; skew 500 ppm
Aug 08 18:34:07 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:07.964169 4686 init.cc:377] Logged in from keytab as oryx/127.27.251.194@KRBTEST.COM (short username oryx)
I20260808 18:34:07.965147 4686 webserver.cc:460] Webserver started at http://127.27.251.194:34049/ using document root <none> and password file <none>
I20260808 18:34:07.965757 4686 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:07.965852 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:07.966099 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:07.968235 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "e9702100462a4b49a6e67bce98d8dddd"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "ee245e14a6e65b609f7461aff96d1ff9"
server_key_iv: "ab0528bb45a9a71a4c70e45c85eb8ecd"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.968834 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "e9702100462a4b49a6e67bce98d8dddd"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "ee245e14a6e65b609f7461aff96d1ff9"
server_key_iv: "ab0528bb45a9a71a4c70e45c85eb8ecd"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.972680 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:34:07.975180 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:07.976409 4686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:07.976599 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "e9702100462a4b49a6e67bce98d8dddd"
format_stamp: "Formatted at 2026-08-08 18:34:07 on dist-test-slave-fd7l"
server_key: "ee245e14a6e65b609f7461aff96d1ff9"
server_key_iv: "ab0528bb45a9a71a4c70e45c85eb8ecd"
server_key_version: "encryptionkey@0"
I20260808 18:34:07.976722 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:08.019760 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:08.023396 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:08.023612 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:08.025277 4686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:08.025349 4686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:08.025424 4686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:08.025456 4686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:08.036978 4686 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:43223
I20260808 18:34:08.037058 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:43223 every 8 connection(s)
I20260808 18:34:08.037992 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:08.040361 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.026009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42261 (local address 127.27.251.254:39133)
0808 18:34:08.026159 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.026163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.027248 (+ 1085us) server_negotiation.cc:407] Connection header received
0808 18:34:08.028288 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.028295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.028368 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.028471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.030279 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.031150 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.033103 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.033334 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.033462 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.036630 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.036635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.036638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.036671 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.038512 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.039069 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.039072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.039073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.039160 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.039398 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.039400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.039401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.039576 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.039648 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.040102 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.040205 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:08.041204 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.026572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.194:42261)
0808 18:34:08.027092 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.027136 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.028022 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.028803 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.028814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.029341 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:08.030076 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.030098 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.031310 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.031314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.032956 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.032972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.033513 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.033519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.033722 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.034322 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:08.034343 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:08.036400 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:08.038648 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.038653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.038667 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.038963 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.039224 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.039226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.039227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.039341 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.039669 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:08.039674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:08.039890 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.040054 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.040288 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
I20260808 18:34:08.042716 4835 heartbeater.cc:347] Connected to a master server at 127.27.251.254:39133
I20260808 18:34:08.042847 4835 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:08.043619 4835 heartbeater.cc:511] Master 127.27.251.254:39133 requested a full tablet report, sending...
I20260808 18:34:08.044796 4451 ts_manager.cc:194] Registered new tserver with Master: e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223)
I20260808 18:34:08.045457 4451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:42261
I20260808 18:34:08.047977 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 4686
I20260808 18:34:08.048112 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:08.048484 28655 external_mini_cluster.cc:1428] Setting key c40e743e8ccc714ab55e4b85d34735d3
I20260808 18:34:08.052636 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.044992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:50343 (local address 127.27.251.254:39133)
0808 18:34:08.045132 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.045136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.045265 (+ 129us) server_negotiation.cc:407] Connection header received
0808 18:34:08.045425 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.045428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.045472 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.045622 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.046105 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.046614 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.047431 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.047669 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.047799 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.048988 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.048991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.048993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.049015 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.050781 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.051220 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.051226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.051230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.051325 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.051586 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.051592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.051597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.051914 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.052026 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.052356 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.052449 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:34:08.052727 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.044899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.194:50343)
0808 18:34:08.045176 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.045190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.045321 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.045634 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.045637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.045878 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:08.045984 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.045991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.046744 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.046747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.047312 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.047320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.047803 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.047807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.047896 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.048428 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:08.048439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:08.048834 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:08.050898 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.050900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.050902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.051105 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.051417 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.051419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.051420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.051496 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.052042 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:08.052047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:08.052164 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.052285 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.052446 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
I20260808 18:34:08.079098 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:39133
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:08.197638 4845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:08.197958 4845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:08.198040 4845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:08.202329 4845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:08.202428 4845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:08.202507 4845 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 18:34:08.202585 4845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:08.208218 4845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.251.254:39133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:08.209429 4845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:08.210564 4845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:08.220556 4854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:08.220769 4845 server_base.cc:1034] running on GCE node
W20260808 18:34:08.220571 4852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:08.220547 4850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:08.221333 4845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:08.222054 4845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:08.223241 4845 hybrid_clock.cc:648] HybridClock initialized: now 1786214048223227 us; error 25 us; skew 500 ppm
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:08.226301 4845 init.cc:377] Logged in from keytab as oryx/127.27.251.195@KRBTEST.COM (short username oryx)
I20260808 18:34:08.227257 4845 webserver.cc:460] Webserver started at http://127.27.251.195:36435/ using document root <none> and password file <none>
I20260808 18:34:08.227829 4845 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:08.227924 4845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:08.228199 4845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:08.230279 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "5188f6e007134d778eefc638755fd693"
format_stamp: "Formatted at 2026-08-08 18:34:08 on dist-test-slave-fd7l"
server_key: "73e6ab811d5170fda314be02be0232f3"
server_key_iv: "27d2c2aae28cc9ce5366fe42a1895129"
server_key_version: "encryptionkey@0"
I20260808 18:34:08.230857 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "5188f6e007134d778eefc638755fd693"
format_stamp: "Formatted at 2026-08-08 18:34:08 on dist-test-slave-fd7l"
server_key: "73e6ab811d5170fda314be02be0232f3"
server_key_iv: "27d2c2aae28cc9ce5366fe42a1895129"
server_key_version: "encryptionkey@0"
I20260808 18:34:08.235060 4845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:08.237644 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:08.238936 4845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:08.239081 4845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "5188f6e007134d778eefc638755fd693"
format_stamp: "Formatted at 2026-08-08 18:34:08 on dist-test-slave-fd7l"
server_key: "73e6ab811d5170fda314be02be0232f3"
server_key_iv: "27d2c2aae28cc9ce5366fe42a1895129"
server_key_version: "encryptionkey@0"
I20260808 18:34:08.239194 4845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:08.252377 4845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:08.256717 4845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:08.256945 4845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:08.258893 4845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:08.258976 4845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:08.259037 4845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:08.259071 4845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:08.270390 4845 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:39279
I20260808 18:34:08.270471 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:39279 every 8 connection(s)
I20260808 18:34:08.271652 4845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:08.274116 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.261655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:41043 (local address 127.27.251.254:39133)
0808 18:34:08.261795 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.261799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.261816 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:34:08.262209 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.262215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.262282 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.262347 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.264271 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.265192 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.267059 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.267393 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.267524 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.270466 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.270471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.270476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.270508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.272133 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.272756 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.272759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.272760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.272842 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.273181 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.273183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.273185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.273365 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.273439 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.273892 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.273993 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:34:08.274951 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.260373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.195:41043)
0808 18:34:08.260911 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.260955 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.261928 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.262833 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.262842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.263366 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:08.264084 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.264107 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.265344 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.265347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.266911 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.266925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.267549 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.267552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.267708 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.268382 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:08.268410 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:08.270283 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:08.272265 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.272274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.272292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.272676 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.272928 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.272930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.272931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.273088 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.273451 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:08.273456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:08.273674 (+ 218us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.273827 (+ 153us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.274088 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":180,"threads_started":1}
I20260808 18:34:08.276326 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 4845
I20260808 18:34:08.276472 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:08.276822 28655 external_mini_cluster.cc:1428] Setting key 59cc81ab377b5ad7893e9428942818d9
I20260808 18:34:08.277149 4990 heartbeater.cc:347] Connected to a master server at 127.27.251.254:39133
I20260808 18:34:08.277380 4990 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:08.278046 4990 heartbeater.cc:511] Master 127.27.251.254:39133 requested a full tablet report, sending...
I20260808 18:34:08.279100 4451 ts_manager.cc:194] Registered new tserver with Master: 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:08.279966 4451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:41043
I20260808 18:34:08.286101 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.278875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:33107 (local address 127.27.251.254:39133)
0808 18:34:08.278990 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.278994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.279260 (+ 266us) server_negotiation.cc:407] Connection header received
0808 18:34:08.279377 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.279379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.279421 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.279502 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.279936 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.280690 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.281651 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.281957 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.282065 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.283076 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.283078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.283080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.283102 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.284667 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.285043 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.285046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.285047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.285193 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.285427 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.285429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.285430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.285596 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.285690 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.285897 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.285988 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:34:08.286134 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.278824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.27.251.195:33107)
0808 18:34:08.279176 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.279194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.279300 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.279527 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.279530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.279700 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:08.279793 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.279798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.280838 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.280842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.281525 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.281537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.282081 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.282084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.282165 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.282606 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:08.282617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:08.282945 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:08.284764 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.284767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.284769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.284959 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.285264 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.285266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.285268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.285348 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.285697 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:08.285699 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:08.285781 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.285861 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.285966 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260808 18:34:08.290882 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:08.317363 4506 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:08.310762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.27.251.254:39133)
0808 18:34:08.310900 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.310904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.310918 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:08.310962 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.310964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.311007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.311085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.311896 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.312414 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.313323 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.313541 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.313623 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.317173 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56586: BlockingRecv error: recv got EOF from 127.0.0.1:56586 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:08.326845 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.27.251.254:39133)
0808 18:34:08.319057 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.319060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.319260 (+ 200us) server_negotiation.cc:407] Connection header received
0808 18:34:08.319352 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.319354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.319395 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.319462 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.320014 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.320564 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.321350 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.321557 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.321634 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.324035 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.324038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.324040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.324067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.325419 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.325811 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.325813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.325815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.325889 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.326151 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.326153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.326154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.326326 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.326430 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.326624 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.326719 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:34:08.330022 4451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:08.333202 4451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:08.348734 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.342803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.27.251.195:39279)
0808 18:34:08.344108 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.344120 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.344145 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:34:08.344267 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.344273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.344446 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.344880 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.345021 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.345915 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.347362 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.347955 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.348098 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.348251 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.348423 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.348574 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":147,"threads_started":1}
I20260808 18:34:08.348743 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.342250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39279 (local address 127.0.0.1:51770)
0808 18:34:08.343526 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.343539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.343746 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.344623 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.344635 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.344648 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.344746 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.344751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.346035 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.346038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.347230 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.347240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.348080 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.348084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.348201 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.348252 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.348380 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.348566 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":170,"thread_start_us":128,"threads_started":1}
I20260808 18:34:08.349772 5017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.342065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:45123 (local address 127.0.0.1:44370)
0808 18:34:08.343057 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.343118 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.343278 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.344126 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.344133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.344161 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.344459 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.344471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.346117 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.346123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.347785 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.347798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.349222 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.349225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.349407 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.349448 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.349527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.349612 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":286,"thread_start_us":96,"threads_started":1}
I20260808 18:34:08.350152 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.343332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.27.251.194:43223)
0808 18:34:08.343848 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.343862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.343892 (+ 30us) server_negotiation.cc:407] Connection header received
0808 18:34:08.344167 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.344178 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.344378 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.344585 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.344838 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.346003 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.347930 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.348842 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.349002 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.349121 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.349697 (+ 576us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.349790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":158,"threads_started":1}
I20260808 18:34:08.350240 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.342266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.27.251.193:45123)
0808 18:34:08.342704 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.342712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.343224 (+ 512us) server_negotiation.cc:407] Connection header received
0808 18:34:08.343454 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.343461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.343854 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.344003 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.344692 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.345926 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.347955 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.348867 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.349010 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.349153 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.349951 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.350060 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":88,"threads_started":1}
I20260808 18:34:08.350416 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.342850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.0.0.1:53188)
0808 18:34:08.343706 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.343722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.343931 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.344574 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.344579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.344603 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.344739 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.344746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.346150 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.346153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.347785 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.347798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.349190 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.349193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.349366 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.349410 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.349488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.349560 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":294,"thread_start_us":100,"threads_started":1}
I20260808 18:34:08.352490 4917 tablet_service.cc:1467] Processing CreateTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (DEFAULT_TABLE table=test-table [id=9b3422eed1254307a7bc349d74860033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.353086 4758 tablet_service.cc:1467] Processing CreateTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (DEFAULT_TABLE table=test-table [id=9b3422eed1254307a7bc349d74860033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.353760 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.354116 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.360076 4600 tablet_service.cc:1467] Processing CreateTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (DEFAULT_TABLE table=test-table [id=9b3422eed1254307a7bc349d74860033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.361172 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.362202 5022 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Bootstrap starting.
I20260808 18:34:08.362807 5023 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Bootstrap starting.
I20260808 18:34:08.364706 5023 tablet_bootstrap.cc:654] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.364794 5022 tablet_bootstrap.cc:654] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.365497 5023 log.cc:824] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:08.365757 5022 log.cc:824] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:08.367565 5023 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: No bootstrap required, opened a new log
I20260808 18:34:08.367806 5023 ts_tablet_manager.cc:1397] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 18:34:08.368028 5022 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: No bootstrap required, opened a new log
I20260808 18:34:08.368211 5022 ts_tablet_manager.cc:1397] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:34:08.369055 5026 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Bootstrap starting.
I20260808 18:34:08.371454 5026 tablet_bootstrap.cc:654] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.371625 5022 raft_consensus.cc:348] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.371888 5022 raft_consensus.cc:374] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.371953 5022 raft_consensus.cc:729] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5188f6e007134d778eefc638755fd693, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.371868 5023 raft_consensus.cc:348] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.372117 5023 raft_consensus.cc:374] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.372182 5023 raft_consensus.cc:729] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.372345 5026 log.cc:824] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:08.372440 5022 consensus_queue.cc:260] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.372658 5023 consensus_queue.cc:260] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.373559 5022 ts_tablet_manager.cc:1428] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:34:08.373637 5023 ts_tablet_manager.cc:1428] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:34:08.373939 4990 heartbeater.cc:503] Master 127.27.251.254:39133 was elected leader, sending a full tablet report...
I20260808 18:34:08.373996 4835 heartbeater.cc:503] Master 127.27.251.254:39133 was elected leader, sending a full tablet report...
I20260808 18:34:08.374831 5026 tablet_bootstrap.cc:492] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: No bootstrap required, opened a new log
I20260808 18:34:08.375109 5026 ts_tablet_manager.cc:1397] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:34:08.379786 5026 raft_consensus.cc:348] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.380028 5026 raft_consensus.cc:374] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.380064 5026 raft_consensus.cc:729] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.380592 5026 consensus_queue.cc:260] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.381460 4674 heartbeater.cc:503] Master 127.27.251.254:39133 was elected leader, sending a full tablet report...
I20260808 18:34:08.381767 5026 ts_tablet_manager.cc:1428] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:34:08.472049 5028 raft_consensus.cc:482] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:08.472309 5028 raft_consensus.cc:504] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.473567 5028 leader_election.cc:288] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223)
I20260808 18:34:08.477581 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.473775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:45123 (local address 127.27.251.195:36503)
0808 18:34:08.473919 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.473936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.474048 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.474395 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.474398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.474452 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.474594 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.474600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.475377 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.475380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.476239 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.476250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.477135 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.477139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.477257 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.477276 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.477366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.477449 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"spinlock_wait_cycles":51328}
I20260808 18:34:08.477581 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.473922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:36503 (local address 127.27.251.193:45123)
0808 18:34:08.474077 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.474085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.474124 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:34:08.474211 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.474217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.474280 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.474344 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.474713 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.475256 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.476367 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.477012 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.477141 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.477196 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.477383 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.477453 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:34:08.478236 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe33c5fa3ff4263b5e7a8d6f44e29f4" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0" is_pre_election: true
I20260808 18:34:08.478336 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.473915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.27.251.195:40145)
0808 18:34:08.474304 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.474315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.474443 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.474747 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.474750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.474772 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.474885 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.474889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.475763 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.475768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.477163 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.477174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.477880 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.477884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.477993 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.478021 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.478124 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.478184 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"spinlock_wait_cycles":1280,"thread_start_us":162,"threads_started":1}
I20260808 18:34:08.478346 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.474099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:40145 (local address 127.27.251.194:43223)
0808 18:34:08.474235 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.474240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.474440 (+ 200us) server_negotiation.cc:407] Connection header received
0808 18:34:08.474547 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.474549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.474599 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.474682 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.475007 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.475644 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.477288 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.477770 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.477894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.477938 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.478148 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.478224 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:34:08.478607 4623 raft_consensus.cc:2454] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 0.
I20260808 18:34:08.479085 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe33c5fa3ff4263b5e7a8d6f44e29f4" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd" is_pre_election: true
I20260808 18:34:08.479139 4865 leader_election.cc:302] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:08.479382 4783 raft_consensus.cc:2454] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 0.
I20260808 18:34:08.479414 5028 raft_consensus.cc:2781] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:08.479509 5028 raft_consensus.cc:482] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:08.479614 5028 raft_consensus.cc:3037] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.480880 5028 raft_consensus.cc:504] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.481312 5028 leader_election.cc:288] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 election: Requested vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223)
I20260808 18:34:08.481703 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe33c5fa3ff4263b5e7a8d6f44e29f4" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd"
I20260808 18:34:08.481745 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe33c5fa3ff4263b5e7a8d6f44e29f4" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0"
I20260808 18:34:08.481829 4783 raft_consensus.cc:3037] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.481848 4623 raft_consensus.cc:3037] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.482900 4623 raft_consensus.cc:2454] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 1.
I20260808 18:34:08.483153 4783 raft_consensus.cc:2454] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 1.
I20260808 18:34:08.483215 4865 leader_election.cc:302] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:08.483368 5028 raft_consensus.cc:2781] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:08.483633 5028 raft_consensus.cc:686] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 1 LEADER]: Becoming Leader. State: Replica: 5188f6e007134d778eefc638755fd693, State: Running, Role: LEADER
I20260808 18:34:08.483963 5028 consensus_queue.cc:237] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } }
I20260808 18:34:08.487042 4451 catalog_manager.cc:5495] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5188f6e007134d778eefc638755fd693 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "5188f6e007134d778eefc638755fd693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } health_report { overall_health: UNKNOWN } } }
I20260808 18:34:08.502372 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.498845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.27.251.195:39279)
0808 18:34:08.498971 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.498975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.498988 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:34:08.499106 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.499108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.499173 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.499263 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:08.499502 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.500093 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.500993 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.501215 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.501294 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.501524 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:08.501955 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:08.502042 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.502190 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.502241 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:34:08.508322 4623 raft_consensus.cc:1261] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Refusing update from remote peer 5188f6e007134d778eefc638755fd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:08.508322 4783 raft_consensus.cc:1261] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Refusing update from remote peer 5188f6e007134d778eefc638755fd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:08.509184 5032 consensus_queue.cc:1035] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:08.509399 5028 consensus_queue.cc:1035] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:08.518047 5036 mvcc.cc:205] Tried to move back new op lower bound from 7316332742682591232 to 7316332742593261568. Current Snapshot: MvccSnapshot[applied={T|T < 7316332742682591232}]
I20260808 18:34:08.526515 4450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:08.526719 4450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:08.529193 4450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Sending DeleteTablet for 3 replicas of tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4
I20260808 18:34:08.530215 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:08 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260808 18:34:08.530226 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:08 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260808 18:34:08.530300 4917 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:08 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:51770
I20260808 18:34:08.530932 5049 tablet_replica.cc:330] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: stopping tablet replica
I20260808 18:34:08.531251 5049 raft_consensus.cc:2229] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:08.531572 5049 raft_consensus.cc:2258] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:08.531522 4450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:08.531965 4450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:08.533079 5049 ts_tablet_manager.cc:1905] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:08.535857 5049 ts_tablet_manager.cc:1918] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:08.535956 5049 log.cc:1196] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/dfe33c5fa3ff4263b5e7a8d6f44e29f4
I20260808 18:34:08.536276 5049 ts_tablet_manager.cc:1939] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P f14be40334fc4070a244ad995b5d88f0: Deleting consensus metadata
I20260808 18:34:08.536670 4600 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7289630b9f488495937d647f4432b3 (DEFAULT_TABLE table=test-table [id=45409bd89a82447fb23351b8a90a0132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.536979 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7289630b9f488495937d647f4432b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.537169 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7289630b9f488495937d647f4432b3 (DEFAULT_TABLE table=test-table [id=45409bd89a82447fb23351b8a90a0132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.537447 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7289630b9f488495937d647f4432b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.537550 4438 catalog_manager.cc:4841] TS f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123): tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (table test-table [id=9b3422eed1254307a7bc349d74860033]) successfully deleted
I20260808 18:34:08.537812 5048 tablet_replica.cc:330] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: stopping tablet replica
I20260808 18:34:08.538111 5048 raft_consensus.cc:2229] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:08.538389 5048 raft_consensus.cc:2258] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:08.538676 5047 tablet_replica.cc:330] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: stopping tablet replica
I20260808 18:34:08.538949 5047 raft_consensus.cc:2229] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:08.539397 4917 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7289630b9f488495937d647f4432b3 (DEFAULT_TABLE table=test-table [id=45409bd89a82447fb23351b8a90a0132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:08.539541 5047 raft_consensus.cc:2258] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:08.539695 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7289630b9f488495937d647f4432b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.540262 5026 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Bootstrap starting.
I20260808 18:34:08.540932 5048 ts_tablet_manager.cc:1905] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:08.541260 5026 tablet_bootstrap.cc:654] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.542348 5022 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Bootstrap starting.
I20260808 18:34:08.542497 5023 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Bootstrap starting.
I20260808 18:34:08.543346 5026 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: No bootstrap required, opened a new log
I20260808 18:34:08.543380 5022 tablet_bootstrap.cc:654] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.543423 5026 ts_tablet_manager.cc:1397] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:08.543601 5023 tablet_bootstrap.cc:654] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.544000 5026 raft_consensus.cc:348] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.544117 5026 raft_consensus.cc:374] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.544173 5026 raft_consensus.cc:729] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.544237 5047 ts_tablet_manager.cc:1905] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:08.544298 5026 consensus_queue.cc:260] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.544557 5026 ts_tablet_manager.cc:1428] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 18:34:08.546137 5023 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: No bootstrap required, opened a new log
I20260808 18:34:08.546257 5023 ts_tablet_manager.cc:1397] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 18:34:08.546731 5023 raft_consensus.cc:348] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.546850 5023 raft_consensus.cc:374] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.546865 5047 ts_tablet_manager.cc:1918] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:08.546891 5023 raft_consensus.cc:729] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.546948 5047 log.cc:1196] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/dfe33c5fa3ff4263b5e7a8d6f44e29f4
I20260808 18:34:08.547037 5023 consensus_queue.cc:260] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.547276 5047 ts_tablet_manager.cc:1939] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P 5188f6e007134d778eefc638755fd693: Deleting consensus metadata
I20260808 18:34:08.547330 5023 ts_tablet_manager.cc:1428] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:34:08.548377 4436 catalog_manager.cc:4841] TS 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279): tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (table test-table [id=9b3422eed1254307a7bc349d74860033]) successfully deleted
I20260808 18:34:08.549010 5048 ts_tablet_manager.cc:1918] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:08.549110 5048 log.cc:1196] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/dfe33c5fa3ff4263b5e7a8d6f44e29f4
I20260808 18:34:08.549432 5048 ts_tablet_manager.cc:1939] T dfe33c5fa3ff4263b5e7a8d6f44e29f4 P e9702100462a4b49a6e67bce98d8dddd: Deleting consensus metadata
I20260808 18:34:08.550618 4436 catalog_manager.cc:4841] TS e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223): tablet dfe33c5fa3ff4263b5e7a8d6f44e29f4 (table test-table [id=9b3422eed1254307a7bc349d74860033]) successfully deleted
I20260808 18:34:08.551276 5022 tablet_bootstrap.cc:492] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: No bootstrap required, opened a new log
I20260808 18:34:08.551373 5022 ts_tablet_manager.cc:1397] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260808 18:34:08.551849 5022 raft_consensus.cc:348] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.551967 5022 raft_consensus.cc:374] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.551999 5022 raft_consensus.cc:729] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5188f6e007134d778eefc638755fd693, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.552121 5022 consensus_queue.cc:260] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.552361 5022 ts_tablet_manager.cc:1428] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:08.589550 5043 raft_consensus.cc:482] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:08.589723 5043 raft_consensus.cc:504] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.590181 5043 leader_election.cc:288] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223)
I20260808 18:34:08.590677 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7289630b9f488495937d647f4432b3" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0" is_pre_election: true
I20260808 18:34:08.590704 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7289630b9f488495937d647f4432b3" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd" is_pre_election: true
I20260808 18:34:08.590857 4783 raft_consensus.cc:2454] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 0.
I20260808 18:34:08.590859 4623 raft_consensus.cc:2454] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 0.
I20260808 18:34:08.591167 4863 leader_election.cc:302] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, e9702100462a4b49a6e67bce98d8dddd; no voters:
I20260808 18:34:08.591341 5043 raft_consensus.cc:2781] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:08.591413 5043 raft_consensus.cc:482] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:08.591464 5043 raft_consensus.cc:3037] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.592568 5043 raft_consensus.cc:504] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.592975 5043 leader_election.cc:288] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 election: Requested vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223)
I20260808 18:34:08.593307 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7289630b9f488495937d647f4432b3" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0"
I20260808 18:34:08.593434 4623 raft_consensus.cc:3037] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.593371 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7289630b9f488495937d647f4432b3" candidate_uuid: "5188f6e007134d778eefc638755fd693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd"
I20260808 18:34:08.593533 4783 raft_consensus.cc:3037] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:08.594525 4783 raft_consensus.cc:2454] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 1.
I20260808 18:34:08.594905 4863 leader_election.cc:302] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, e9702100462a4b49a6e67bce98d8dddd; no voters:
I20260808 18:34:08.595126 4623 raft_consensus.cc:2454] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5188f6e007134d778eefc638755fd693 in term 1.
I20260808 18:34:08.595140 5043 raft_consensus.cc:2781] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:08.595228 5043 raft_consensus.cc:686] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 1 LEADER]: Becoming Leader. State: Replica: 5188f6e007134d778eefc638755fd693, State: Running, Role: LEADER
I20260808 18:34:08.595369 5043 consensus_queue.cc:237] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.597157 4451 catalog_manager.cc:5495] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5188f6e007134d778eefc638755fd693 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "5188f6e007134d778eefc638755fd693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } health_report { overall_health: HEALTHY } } }
Aug 08 18:34:08 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214047, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.254@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:08.635277 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.626258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:39133 (local address 127.0.0.1:56610)
0808 18:34:08.626573 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.626597 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.626698 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.626977 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.626979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.627361 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:08.627459 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.627465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.628489 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.628491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.629123 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.629140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.629743 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.629746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.629809 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.630402 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:08.630414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:08.631884 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:08.633760 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.633769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.633774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.633998 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.634280 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:08.634282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:08.634284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:08.634360 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:08.634827 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:08.634832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:08.634930 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 18:34:08.635018 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:34:08.635137 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260808 18:34:08.635409 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.626373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.27.251.254:39133)
0808 18:34:08.626540 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.626545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.626673 (+ 128us) server_negotiation.cc:407] Connection header received
0808 18:34:08.626785 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.626788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.626843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.626943 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:08.627581 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.628386 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.629248 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.629614 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.629750 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:08.632022 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:08.632026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:08.632029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:08.632055 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:08.633642 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.634093 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.634097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.634099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.634186 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:08.634456 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:08.634460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:08.634462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:08.634725 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:08.634830 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:08.635101 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 18:34:08.635219 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:34:08.639202 4451 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:56610:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 18:34:08.639937 4451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:08.645253 4600 tablet_service.cc:1467] Processing CreateTablet for tablet 13a75bef447844bbb7a7317664cdf106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de9510d782e4616ab3964a45432bf2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:34:08.645424 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 13a75bef447844bbb7a7317664cdf106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de9510d782e4616ab3964a45432bf2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:34:08.645717 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a75bef447844bbb7a7317664cdf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.645870 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a75bef447844bbb7a7317664cdf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.648232 5023 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: Bootstrap starting.
I20260808 18:34:08.648592 4917 tablet_service.cc:1467] Processing CreateTablet for tablet 13a75bef447844bbb7a7317664cdf106 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1de9510d782e4616ab3964a45432bf2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 18:34:08.648802 5026 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: Bootstrap starting.
I20260808 18:34:08.648962 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a75bef447844bbb7a7317664cdf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:08.649271 5023 tablet_bootstrap.cc:654] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.649760 5026 tablet_bootstrap.cc:654] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.651001 5026 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: No bootstrap required, opened a new log
I20260808 18:34:08.651098 5026 ts_tablet_manager.cc:1397] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:08.651362 5023 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: No bootstrap required, opened a new log
I20260808 18:34:08.651440 5023 ts_tablet_manager.cc:1397] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:08.651554 5026 raft_consensus.cc:348] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.651670 5026 raft_consensus.cc:374] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.651712 5026 raft_consensus.cc:729] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.651856 5026 consensus_queue.cc:260] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.651885 5023 raft_consensus.cc:348] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.651996 5023 raft_consensus.cc:374] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.652030 5023 raft_consensus.cc:729] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.652159 5023 consensus_queue.cc:260] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.652297 5030 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:08.652424 5030 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:08.652441 5023 ts_tablet_manager.cc:1428] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 18:34:08.652598 5029 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:08.652743 5029 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:08.653241 5022 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: Bootstrap starting.
I20260808 18:34:08.654220 5022 tablet_bootstrap.cc:654] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:08.654448 5026 ts_tablet_manager.cc:1428] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:34:08.655691 5022 tablet_bootstrap.cc:492] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: No bootstrap required, opened a new log
I20260808 18:34:08.655787 5022 ts_tablet_manager.cc:1397] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:34:08.656217 5022 raft_consensus.cc:348] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.656324 5022 raft_consensus.cc:374] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:08.656364 5022 raft_consensus.cc:729] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5188f6e007134d778eefc638755fd693, State: Initialized, Role: FOLLOWER
I20260808 18:34:08.656497 5022 consensus_queue.cc:260] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.656769 5043 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:08.656893 5043 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:08.657126 5022 ts_tablet_manager.cc:1428] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260808 18:34:08.773337 4991 tablet.cc:2369] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:34:08.780323 4675 tablet.cc:2369] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:34:08.789290 4836 tablet.cc:2369] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:34:08.995573 5029 raft_consensus.cc:482] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:08.995718 5029 raft_consensus.cc:504] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:08.997001 5029 leader_election.cc:288] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.001027 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.997413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:60635 (local address 127.27.251.195:39279)
0808 18:34:08.997572 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.997577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.997730 (+ 153us) server_negotiation.cc:407] Connection header received
0808 18:34:08.997917 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.997919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.997968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.998060 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.998372 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.998916 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.999969 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.000380 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.000547 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.000605 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.000829 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.000906 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:34:09.001111 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.997262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:45123 (local address 127.27.251.194:41403)
0808 18:34:08.997764 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.997776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.997898 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.998184 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.998187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.998203 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.998344 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.998349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.999194 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.999200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.999950 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.999959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.000672 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.000676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.000785 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.000809 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.000887 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.000954 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260808 18:34:09.001096 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.997376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:41403 (local address 127.27.251.193:45123)
0808 18:34:08.997665 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:08.997671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:08.997857 (+ 186us) server_negotiation.cc:407] Connection header received
0808 18:34:08.998018 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:08.998022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:08.998072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:08.998175 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:08.998463 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.999074 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.000082 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.000497 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.000683 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.000735 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.000905 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.000978 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260808 18:34:09.001448 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:08.997260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39279 (local address 127.27.251.194:60635)
0808 18:34:08.997630 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:08.997644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:08.997789 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:08.998076 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:08.998079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:08.998097 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:08.998251 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.998256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:08.999042 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:08.999047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:08.999831 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:08.999840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.000547 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.000550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.000647 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.000676 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.001199 (+ 523us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.001312 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":19,"thread_start_us":92,"threads_started":1}
I20260808 18:34:09.001562 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a75bef447844bbb7a7317664cdf106" candidate_uuid: "e9702100462a4b49a6e67bce98d8dddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0" is_pre_election: true
I20260808 18:34:09.001744 4623 raft_consensus.cc:2454] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9702100462a4b49a6e67bce98d8dddd in term 0.
I20260808 18:34:09.002002 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a75bef447844bbb7a7317664cdf106" candidate_uuid: "e9702100462a4b49a6e67bce98d8dddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693" is_pre_election: true
I20260808 18:34:09.002154 4705 leader_election.cc:302] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9702100462a4b49a6e67bce98d8dddd, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:09.002257 4937 raft_consensus.cc:2454] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9702100462a4b49a6e67bce98d8dddd in term 0.
I20260808 18:34:09.002437 5029 raft_consensus.cc:2781] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:09.002498 5029 raft_consensus.cc:482] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:09.002547 5029 raft_consensus.cc:3037] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.003549 5029 raft_consensus.cc:504] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.003949 5029 leader_election.cc:288] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [CANDIDATE]: Term 1 election: Requested vote from peers f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.004309 4623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a75bef447844bbb7a7317664cdf106" candidate_uuid: "e9702100462a4b49a6e67bce98d8dddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14be40334fc4070a244ad995b5d88f0"
I20260808 18:34:09.004360 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a75bef447844bbb7a7317664cdf106" candidate_uuid: "e9702100462a4b49a6e67bce98d8dddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693"
I20260808 18:34:09.004426 4623 raft_consensus.cc:3037] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.004557 4937 raft_consensus.cc:3037] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.005378 4623 raft_consensus.cc:2454] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9702100462a4b49a6e67bce98d8dddd in term 1.
I20260808 18:34:09.005682 4937 raft_consensus.cc:2454] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9702100462a4b49a6e67bce98d8dddd in term 1.
I20260808 18:34:09.006022 4703 leader_election.cc:302] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, e9702100462a4b49a6e67bce98d8dddd; no voters:
I20260808 18:34:09.006325 5029 raft_consensus.cc:2781] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:09.006488 5029 raft_consensus.cc:686] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [term 1 LEADER]: Becoming Leader. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Running, Role: LEADER
I20260808 18:34:09.006731 5029 consensus_queue.cc:237] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.008793 5067 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: TxnStatusTablet state changed. Reason: New leader e9702100462a4b49a6e67bce98d8dddd. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.008916 5067 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:34:09.009233 5069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 18:34:09.009409 5069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 18:34:09.009485 4450 catalog_manager.cc:5495] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd reported cstate change: term changed from 0 to 1, leader changed from <none> to e9702100462a4b49a6e67bce98d8dddd (127.27.251.194). New cstate: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } health_report { overall_health: UNKNOWN } } }
I20260808 18:34:09.022563 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.018391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.27.251.194:43223)
0808 18:34:09.018617 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.018621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.018636 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:09.018719 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.018722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.018783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.018901 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.019170 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.020043 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.021063 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.021327 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.021436 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.021735 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.022155 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.022282 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.022355 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.022406 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260808 18:34:09.022547 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.018308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.0.0.1:53190)
0808 18:34:09.018446 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.018463 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.018570 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.018928 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.018930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.018937 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:34:09.019025 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.019031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.020161 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.020164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.020920 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.020931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.021448 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.021451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.021541 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.021584 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.022256 (+ 672us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.022273 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.022345 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.022431 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 18:34:09.025454 4623 raft_consensus.cc:1261] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Refusing update from remote peer e9702100462a4b49a6e67bce98d8dddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.025734 4937 raft_consensus.cc:1261] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Refusing update from remote peer e9702100462a4b49a6e67bce98d8dddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.026000 5067 consensus_queue.cc:1035] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.026310 5029 consensus_queue.cc:1035] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.027565 5030 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: TxnStatusTablet state changed. Reason: New leader e9702100462a4b49a6e67bce98d8dddd. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.027681 5030 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:09.028005 5043 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: TxnStatusTablet state changed. Reason: New leader e9702100462a4b49a6e67bce98d8dddd. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.028146 5043 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:09.030035 5029 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.030156 5029 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:34:09.031957 5043 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.032066 5043 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P 5188f6e007134d778eefc638755fd693: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:09.032398 5077 mvcc.cc:205] Tried to move back new op lower bound from 7316332744803094528 to 7316332744734380032. Current Snapshot: MvccSnapshot[applied={T|T < 7316332744803094528}]
I20260808 18:34:09.037812 5067 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.037925 5067 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P e9702100462a4b49a6e67bce98d8dddd: This TxnStatusTablet replica's current role is: LEADER
I20260808 18:34:09.038589 5030 tablet_replica.cc:439] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9702100462a4b49a6e67bce98d8dddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } } }
I20260808 18:34:09.038695 5030 tablet_replica.cc:442] T 13a75bef447844bbb7a7317664cdf106 P f14be40334fc4070a244ad995b5d88f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 18:34:09.039327 5075 mvcc.cc:205] Tried to move back new op lower bound from 7316332744803094528 to 7316332744734380032. Current Snapshot: MvccSnapshot[applied={T|T < 7316332744803094528}]
I20260808 18:34:09.048352 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.044222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:56511 (local address 127.27.251.194:43223)
0808 18:34:09.044365 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.044371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.044814 (+ 443us) server_negotiation.cc:407] Connection header received
0808 18:34:09.044973 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.044976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.045031 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.045116 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.045361 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.046047 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.046933 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.047161 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.047296 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.047733 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.047833 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.047899 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.048104 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.048180 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:34:09.048767 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.044177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.27.251.195:56511)
0808 18:34:09.044708 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.044755 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.044882 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.045128 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.045131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.045139 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:34:09.045242 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.045248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.046155 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.046160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.046805 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.046814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.047283 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.047287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.047380 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.047435 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.047982 (+ 547us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.047988 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.048567 (+ 579us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.048660 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":127,"threads_started":1}
I20260808 18:34:09.056067 4783 raft_consensus.cc:1261] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Refusing update from remote peer 5188f6e007134d778eefc638755fd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.056067 4623 raft_consensus.cc:1261] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Refusing update from remote peer 5188f6e007134d778eefc638755fd693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.056461 5032 consensus_queue.cc:1035] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.056634 5043 consensus_queue.cc:1035] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.077868 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.073941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:38739 (local address 127.27.251.195:39279)
0808 18:34:09.074055 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.074059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.074301 (+ 242us) server_negotiation.cc:407] Connection header received
0808 18:34:09.074418 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.074421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.074469 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.074548 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.074788 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.075566 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.076723 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.076953 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.077084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.077406 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.077522 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.077593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.077682 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.077733 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 18:34:09.077890 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.073894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39279 (local address 127.27.251.194:38739)
0808 18:34:09.074218 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.074231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.074329 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.074565 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.074567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.074573 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:34:09.074690 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.074695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.075666 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.075670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.076602 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.076612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.077084 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.077088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.077185 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.077243 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.077600 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.077609 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.077672 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.077747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":121,"threads_started":1}
I20260808 18:34:09.101816 4450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.101969 4450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.104146 4450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Sending DeleteTablet for 3 replicas of tablet 1f7289630b9f488495937d647f4432b3
I20260808 18:34:09.104722 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7289630b9f488495937d647f4432b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260808 18:34:09.104846 4917 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7289630b9f488495937d647f4432b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:51770
I20260808 18:34:09.105186 5094 tablet_replica.cc:330] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: stopping tablet replica
I20260808 18:34:09.105206 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7289630b9f488495937d647f4432b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260808 18:34:09.105446 5094 raft_consensus.cc:2229] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:09.105571 5093 tablet_replica.cc:330] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: stopping tablet replica
I20260808 18:34:09.105693 5093 raft_consensus.cc:2229] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.105831 5093 raft_consensus.cc:2258] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.105831 5094 raft_consensus.cc:2258] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.106596 5093 ts_tablet_manager.cc:1905] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.106596 5094 ts_tablet_manager.cc:1905] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.109647 5093 ts_tablet_manager.cc:1918] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:34:09.109750 5093 log.cc:1196] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/1f7289630b9f488495937d647f4432b3
I20260808 18:34:09.109906 5094 ts_tablet_manager.cc:1918] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:34:09.110059 5093 ts_tablet_manager.cc:1939] T 1f7289630b9f488495937d647f4432b3 P f14be40334fc4070a244ad995b5d88f0: Deleting consensus metadata
I20260808 18:34:09.110076 5094 log.cc:1196] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/1f7289630b9f488495937d647f4432b3
I20260808 18:34:09.110491 5094 ts_tablet_manager.cc:1939] T 1f7289630b9f488495937d647f4432b3 P 5188f6e007134d778eefc638755fd693: Deleting consensus metadata
I20260808 18:34:09.110915 4438 catalog_manager.cc:4841] TS f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123): tablet 1f7289630b9f488495937d647f4432b3 (table test-table [id=45409bd89a82447fb23351b8a90a0132]) successfully deleted
I20260808 18:34:09.111443 4436 catalog_manager.cc:4841] TS 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279): tablet 1f7289630b9f488495937d647f4432b3 (table test-table [id=45409bd89a82447fb23351b8a90a0132]) successfully deleted
I20260808 18:34:09.112707 5095 tablet_replica.cc:330] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: stopping tablet replica
I20260808 18:34:09.112879 5095 raft_consensus.cc:2229] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.113020 5095 raft_consensus.cc:2258] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.114095 5095 ts_tablet_manager.cc:1905] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.117018 5095 ts_tablet_manager.cc:1918] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:34:09.117103 5095 log.cc:1196] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/1f7289630b9f488495937d647f4432b3
I20260808 18:34:09.117393 5095 ts_tablet_manager.cc:1939] T 1f7289630b9f488495937d647f4432b3 P e9702100462a4b49a6e67bce98d8dddd: Deleting consensus metadata
I20260808 18:34:09.118127 4436 catalog_manager.cc:4841] TS e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223): tablet 1f7289630b9f488495937d647f4432b3 (table test-table [id=45409bd89a82447fb23351b8a90a0132]) successfully deleted
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[4405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:09.135330 4506 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:09.128257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.27.251.254:39133)
0808 18:34:09.128403 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.128407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.128553 (+ 146us) server_negotiation.cc:407] Connection header received
0808 18:34:09.128722 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.128726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.128781 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.128877 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:09.129557 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.130385 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.131275 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.131600 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.131715 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.135200 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56620: BlockingRecv error: recv got EOF from 127.0.0.1:56620 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[4405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM
I20260808 18:34:09.146725 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.136855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.27.251.254:39133)
0808 18:34:09.136969 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.136973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.137099 (+ 126us) server_negotiation.cc:407] Connection header received
0808 18:34:09.137185 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.137188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.137235 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.137314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:09.137907 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.138666 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.139429 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.139769 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.139881 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.142507 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:09.142514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:09.142517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:09.142544 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:09.144746 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:09.145270 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:09.145274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:09.145276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:09.145397 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:09.145729 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:09.145733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:09.145735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:09.145932 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:09.146074 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.146325 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.146441 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:34:09.149674 4450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:09.150171 4450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:09.155289 4600 tablet_service.cc:1467] Processing CreateTablet for tablet f95946298e7d4800a1e61f43ed1202a3 (DEFAULT_TABLE table=test-table [id=c446248e59004afb8cab93ba0e6102de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.155416 4917 tablet_service.cc:1467] Processing CreateTablet for tablet f95946298e7d4800a1e61f43ed1202a3 (DEFAULT_TABLE table=test-table [id=c446248e59004afb8cab93ba0e6102de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.155645 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95946298e7d4800a1e61f43ed1202a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.155728 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95946298e7d4800a1e61f43ed1202a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.158098 5109 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Bootstrap starting.
I20260808 18:34:09.158454 4758 tablet_service.cc:1467] Processing CreateTablet for tablet f95946298e7d4800a1e61f43ed1202a3 (DEFAULT_TABLE table=test-table [id=c446248e59004afb8cab93ba0e6102de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.158761 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95946298e7d4800a1e61f43ed1202a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.158934 5110 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Bootstrap starting.
I20260808 18:34:09.159564 5109 tablet_bootstrap.cc:654] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.159883 5110 tablet_bootstrap.cc:654] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.161567 5109 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: No bootstrap required, opened a new log
I20260808 18:34:09.161681 5109 ts_tablet_manager.cc:1397] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 18:34:09.162110 5114 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Bootstrap starting.
I20260808 18:34:09.162200 5109 raft_consensus.cc:348] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.162331 5109 raft_consensus.cc:374] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.162372 5109 raft_consensus.cc:729] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.162503 5109 consensus_queue.cc:260] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.162806 5109 ts_tablet_manager.cc:1428] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:09.163115 5114 tablet_bootstrap.cc:654] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.164283 5110 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: No bootstrap required, opened a new log
I20260808 18:34:09.164356 5110 ts_tablet_manager.cc:1397] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 18:34:09.164811 5110 raft_consensus.cc:348] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.164933 5110 raft_consensus.cc:374] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.164976 5110 raft_consensus.cc:729] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5188f6e007134d778eefc638755fd693, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.165104 5110 consensus_queue.cc:260] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.165498 5110 ts_tablet_manager.cc:1428] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:09.165699 5114 tablet_bootstrap.cc:492] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: No bootstrap required, opened a new log
I20260808 18:34:09.165793 5114 ts_tablet_manager.cc:1397] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 18:34:09.166299 5114 raft_consensus.cc:348] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.166420 5114 raft_consensus.cc:374] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.166462 5114 raft_consensus.cc:729] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.166605 5114 consensus_queue.cc:260] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.167289 5114 ts_tablet_manager.cc:1428] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:09.335162 5030 raft_consensus.cc:482] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:09.335318 5030 raft_consensus.cc:504] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.336676 5030 leader_election.cc:288] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.341068 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.337528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:54581 (local address 127.27.251.195:39279)
0808 18:34:09.337688 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.337692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.337810 (+ 118us) server_negotiation.cc:407] Connection header received
0808 18:34:09.337900 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.337902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.337952 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.338035 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:09.338343 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.338890 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340013 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.340423 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340575 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.340643 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.340852 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.340937 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 18:34:09.341074 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.337378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:39279 (local address 127.27.251.193:54581)
0808 18:34:09.337689 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.337703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.337816 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.338056 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.338058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.338074 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:09.338226 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.338233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.339022 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.339025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.339880 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.339890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.340588 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.340705 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.340737 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.340811 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.340899 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260808 18:34:09.341095 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.336926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.27.251.193:44979)
0808 18:34:09.337241 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.337257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.337378 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.337862 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.337865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.337891 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:09.338044 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.338053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.338898 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.338901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.339900 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.339911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.340658 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.340764 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.340782 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.340880 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.340946 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":83,"thread_start_us":143,"threads_started":1}
I20260808 18:34:09.341138 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.337064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:44979 (local address 127.27.251.194:43223)
0808 18:34:09.337217 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.337221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.337355 (+ 134us) server_negotiation.cc:407] Connection header received
0808 18:34:09.337665 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.337669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.337734 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.337818 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:09.338182 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.338774 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340061 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.340463 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.340655 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.340711 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.340915 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.341005 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:34:09.341583 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95946298e7d4800a1e61f43ed1202a3" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd" is_pre_election: true
I20260808 18:34:09.341739 4783 raft_consensus.cc:2454] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 0.
I20260808 18:34:09.341737 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95946298e7d4800a1e61f43ed1202a3" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693" is_pre_election: true
I20260808 18:34:09.341872 4937 raft_consensus.cc:2454] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 0.
I20260808 18:34:09.342134 4551 leader_election.cc:302] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9702100462a4b49a6e67bce98d8dddd, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:09.342427 5030 raft_consensus.cc:2781] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:09.342526 5030 raft_consensus.cc:482] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:09.342586 5030 raft_consensus.cc:3037] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.343904 5030 raft_consensus.cc:504] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.344331 5030 leader_election.cc:288] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 election: Requested vote from peers e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.344775 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95946298e7d4800a1e61f43ed1202a3" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693"
I20260808 18:34:09.344875 4937 raft_consensus.cc:3037] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.345921 4937 raft_consensus.cc:2454] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 1.
I20260808 18:34:09.346212 4551 leader_election.cc:302] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5188f6e007134d778eefc638755fd693, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:09.346187 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95946298e7d4800a1e61f43ed1202a3" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd"
I20260808 18:34:09.346345 4783 raft_consensus.cc:3037] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.346388 5030 raft_consensus.cc:2781] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:09.346658 5030 raft_consensus.cc:686] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 1 LEADER]: Becoming Leader. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Running, Role: LEADER
I20260808 18:34:09.346948 5030 consensus_queue.cc:237] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.347467 4783 raft_consensus.cc:2454] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 1.
I20260808 18:34:09.349485 4451 catalog_manager.cc:5495] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f14be40334fc4070a244ad995b5d88f0 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "f14be40334fc4070a244ad995b5d88f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } health_report { overall_health: UNKNOWN } } }
I20260808 18:34:09.400950 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.397160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.27.251.193:45123)
0808 18:34:09.397328 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.397333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.397348 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:09.397473 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.397475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.397534 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.397680 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.397958 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.398488 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.399373 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.399603 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.399728 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.400034 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.400545 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.400693 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.400736 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.400798 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 18:34:09.403944 4783 raft_consensus.cc:1261] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Refusing update from remote peer f14be40334fc4070a244ad995b5d88f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.403957 4937 raft_consensus.cc:1261] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Refusing update from remote peer f14be40334fc4070a244ad995b5d88f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.404562 5030 consensus_queue.cc:1035] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.404768 5119 consensus_queue.cc:1035] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.417258 4451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.417429 4451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.419780 4451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Sending DeleteTablet for 3 replicas of tablet f95946298e7d4800a1e61f43ed1202a3
I20260808 18:34:09.420400 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet f95946298e7d4800a1e61f43ed1202a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260808 18:34:09.420388 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet f95946298e7d4800a1e61f43ed1202a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260808 18:34:09.420410 4917 tablet_service.cc:1514] Processing DeleteTablet for tablet f95946298e7d4800a1e61f43ed1202a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:51770
I20260808 18:34:09.420599 5093 tablet_replica.cc:330] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: stopping tablet replica
I20260808 18:34:09.420619 5094 tablet_replica.cc:330] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: stopping tablet replica
I20260808 18:34:09.420643 5095 tablet_replica.cc:330] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: stopping tablet replica
I20260808 18:34:09.420725 5094 raft_consensus.cc:2229] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.420720 5093 raft_consensus.cc:2229] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:09.420760 5095 raft_consensus.cc:2229] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.420845 5094 raft_consensus.cc:2258] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.420893 5095 raft_consensus.cc:2258] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.421015 5093 raft_consensus.cc:2258] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.421347 4451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 18:34:09.421604 5094 ts_tablet_manager.cc:1905] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:34:09.421770 4451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:09.421931 5093 ts_tablet_manager.cc:1905] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.424321 5094 ts_tablet_manager.cc:1918] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:09.424321 5093 ts_tablet_manager.cc:1918] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:09.424401 5094 log.cc:1196] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/f95946298e7d4800a1e61f43ed1202a3
I20260808 18:34:09.424401 5093 log.cc:1196] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/f95946298e7d4800a1e61f43ed1202a3
I20260808 18:34:09.424698 5094 ts_tablet_manager.cc:1939] T f95946298e7d4800a1e61f43ed1202a3 P 5188f6e007134d778eefc638755fd693: Deleting consensus metadata
I20260808 18:34:09.425375 4436 catalog_manager.cc:4841] TS 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279): tablet f95946298e7d4800a1e61f43ed1202a3 (table test-table [id=c446248e59004afb8cab93ba0e6102de]) successfully deleted
I20260808 18:34:09.425657 5093 ts_tablet_manager.cc:1939] T f95946298e7d4800a1e61f43ed1202a3 P f14be40334fc4070a244ad995b5d88f0: Deleting consensus metadata
I20260808 18:34:09.426317 4438 catalog_manager.cc:4841] TS f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123): tablet f95946298e7d4800a1e61f43ed1202a3 (table test-table [id=c446248e59004afb8cab93ba0e6102de]) successfully deleted
I20260808 18:34:09.426882 4600 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 (DEFAULT_TABLE table=test-table [id=41b778e22475474f9f3eeaf2bda3e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.427076 4917 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 (DEFAULT_TABLE table=test-table [id=41b778e22475474f9f3eeaf2bda3e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.427141 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d490ea7f54f088f160eb2dfa69a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.427302 4917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d490ea7f54f088f160eb2dfa69a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.429482 5110 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Bootstrap starting.
I20260808 18:34:09.429492 5109 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: Bootstrap starting.
I20260808 18:34:09.430398 5110 tablet_bootstrap.cc:654] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.430418 5109 tablet_bootstrap.cc:654] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.431129 4758 tablet_service.cc:1467] Processing CreateTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 (DEFAULT_TABLE table=test-table [id=41b778e22475474f9f3eeaf2bda3e2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:09.431409 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2d490ea7f54f088f160eb2dfa69a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:09.431723 5095 ts_tablet_manager.cc:1905] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.432255 5110 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: No bootstrap required, opened a new log
I20260808 18:34:09.432325 5110 ts_tablet_manager.cc:1397] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 18:34:09.432744 5110 raft_consensus.cc:348] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.432850 5110 raft_consensus.cc:374] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.432883 5110 raft_consensus.cc:729] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5188f6e007134d778eefc638755fd693, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.432997 5110 consensus_queue.cc:260] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.433256 5110 ts_tablet_manager.cc:1428] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:09.433789 5095 ts_tablet_manager.cc:1918] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:09.433866 5095 log.cc:1196] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/f95946298e7d4800a1e61f43ed1202a3
I20260808 18:34:09.434151 5095 ts_tablet_manager.cc:1939] T f95946298e7d4800a1e61f43ed1202a3 P e9702100462a4b49a6e67bce98d8dddd: Deleting consensus metadata
I20260808 18:34:09.434828 5114 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Bootstrap starting.
I20260808 18:34:09.434901 4436 catalog_manager.cc:4841] TS e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223): tablet f95946298e7d4800a1e61f43ed1202a3 (table test-table [id=c446248e59004afb8cab93ba0e6102de]) successfully deleted
I20260808 18:34:09.435653 5114 tablet_bootstrap.cc:654] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:09.436200 5109 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: No bootstrap required, opened a new log
I20260808 18:34:09.436275 5109 ts_tablet_manager.cc:1397] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260808 18:34:09.436764 5114 tablet_bootstrap.cc:492] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: No bootstrap required, opened a new log
I20260808 18:34:09.436733 5109 raft_consensus.cc:348] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.436823 5114 ts_tablet_manager.cc:1397] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:09.436839 5109 raft_consensus.cc:374] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.436870 5109 raft_consensus.cc:729] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.436985 5109 consensus_queue.cc:260] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.437268 5109 ts_tablet_manager.cc:1428] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 18:34:09.437207 5114 raft_consensus.cc:348] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.437326 5114 raft_consensus.cc:374] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:09.437366 5114 raft_consensus.cc:729] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9702100462a4b49a6e67bce98d8dddd, State: Initialized, Role: FOLLOWER
I20260808 18:34:09.437480 5114 consensus_queue.cc:260] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.437757 5114 ts_tablet_manager.cc:1428] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:34:09.516644 5030 raft_consensus.cc:482] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:09.516812 5030 raft_consensus.cc:504] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.517331 5030 leader_election.cc:288] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.517894 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d490ea7f54f088f160eb2dfa69a02" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd" is_pre_election: true
I20260808 18:34:09.518061 4783 raft_consensus.cc:2454] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 0.
I20260808 18:34:09.518011 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d490ea7f54f088f160eb2dfa69a02" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693" is_pre_election: true
I20260808 18:34:09.518194 4937 raft_consensus.cc:2454] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 0.
I20260808 18:34:09.518501 4551 leader_election.cc:302] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9702100462a4b49a6e67bce98d8dddd, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:09.518749 5030 raft_consensus.cc:2781] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:09.518853 5030 raft_consensus.cc:482] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:09.518939 5030 raft_consensus.cc:3037] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.520088 5030 raft_consensus.cc:504] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.520445 5030 leader_election.cc:288] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 election: Requested vote from peers e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223), 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279)
I20260808 18:34:09.520910 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d490ea7f54f088f160eb2dfa69a02" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9702100462a4b49a6e67bce98d8dddd"
I20260808 18:34:09.520938 4937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2d490ea7f54f088f160eb2dfa69a02" candidate_uuid: "f14be40334fc4070a244ad995b5d88f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5188f6e007134d778eefc638755fd693"
I20260808 18:34:09.521068 4937 raft_consensus.cc:3037] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.521062 4783 raft_consensus.cc:3037] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:09.522050 4783 raft_consensus.cc:2454] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 1.
I20260808 18:34:09.522413 4551 leader_election.cc:302] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9702100462a4b49a6e67bce98d8dddd, f14be40334fc4070a244ad995b5d88f0; no voters:
I20260808 18:34:09.522497 4937 raft_consensus.cc:2454] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f14be40334fc4070a244ad995b5d88f0 in term 1.
I20260808 18:34:09.522672 5030 raft_consensus.cc:2781] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:09.523025 5030 raft_consensus.cc:686] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 1 LEADER]: Becoming Leader. State: Replica: f14be40334fc4070a244ad995b5d88f0, State: Running, Role: LEADER
I20260808 18:34:09.523216 5030 consensus_queue.cc:237] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } }
I20260808 18:34:09.525547 4450 catalog_manager.cc:5495] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f14be40334fc4070a244ad995b5d88f0 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "f14be40334fc4070a244ad995b5d88f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f14be40334fc4070a244ad995b5d88f0" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 45123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 } health_report { overall_health: UNKNOWN } } }
I20260808 18:34:09.570464 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.566409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:50741 (local address 127.27.251.194:43223)
0808 18:34:09.566532 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.566536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.566830 (+ 294us) server_negotiation.cc:407] Connection header received
0808 18:34:09.566998 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.567000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.567048 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.567111 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.567465 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.567962 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.569396 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.569607 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.569686 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.570036 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.570105 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.570177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.570304 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.570353 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:34:09.570556 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.566330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43223 (local address 127.27.251.193:50741)
0808 18:34:09.566738 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.566755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.566883 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.567184 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.567188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.567197 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:34:09.567352 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.567360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.568087 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.568091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.569276 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.569289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.569733 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.569737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.569841 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.569896 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.570208 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.570215 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.570283 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.570372 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
I20260808 18:34:09.576800 4783 raft_consensus.cc:1261] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Refusing update from remote peer f14be40334fc4070a244ad995b5d88f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.576804 4937 raft_consensus.cc:1261] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Refusing update from remote peer f14be40334fc4070a244ad995b5d88f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:09.577277 5030 consensus_queue.cc:1035] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9702100462a4b49a6e67bce98d8dddd" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.577549 5124 consensus_queue.cc:1035] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5188f6e007134d778eefc638755fd693" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:09.596963 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.593067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:51007 (local address 127.27.251.193:45123)
0808 18:34:09.593197 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:09.593202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:09.593343 (+ 141us) server_negotiation.cc:407] Connection header received
0808 18:34:09.593495 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:09.593499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:09.593552 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:09.593630 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:09.593975 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.594488 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.595798 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.596050 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.596145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.596419 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.596503 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.596603 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:09.596735 (+ 132us) server_negotiation.cc:299] Negotiation successful
0808 18:34:09.596797 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:34:09.597368 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.593007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:45123 (local address 127.27.251.194:51007)
0808 18:34:09.593256 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.593270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:09.593393 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:09.593744 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:09.593748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:09.593757 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 18:34:09.593866 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.593873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.594593 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.594596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.595671 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:09.595681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:09.596166 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:09.596170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:09.596272 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:09.596307 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 18:34:09.596606 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 18:34:09.596613 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 18:34:09.597111 (+ 498us) client_negotiation.cc:240] Negotiation successful
0808 18:34:09.597207 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260808 18:34:09.623736 4450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.623898 4450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:09.625871 4450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a50b8d1e9649e6bbf7140b9968e249: Sending DeleteTablet for 3 replicas of tablet ea2d490ea7f54f088f160eb2dfa69a02
I20260808 18:34:09.626375 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:44370
I20260808 18:34:09.626459 4758 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260808 18:34:09.626586 4917 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2d490ea7f54f088f160eb2dfa69a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:09 UTC) from {username='oryx', principal='oryx/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:51770
I20260808 18:34:09.626684 5093 tablet_replica.cc:330] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: stopping tablet replica
I20260808 18:34:09.626693 5095 tablet_replica.cc:330] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: stopping tablet replica
I20260808 18:34:09.626770 5094 tablet_replica.cc:330] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: stopping tablet replica
I20260808 18:34:09.626818 5093 raft_consensus.cc:2229] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:09.626828 5095 raft_consensus.cc:2229] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.626878 5094 raft_consensus.cc:2229] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:09.626950 5095 raft_consensus.cc:2258] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.626976 5094 raft_consensus.cc:2258] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.627009 5093 raft_consensus.cc:2258] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:09.627761 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 4533
I20260808 18:34:09.627769 5093 ts_tablet_manager.cc:1905] T ea2d490ea7f54f088f160eb2dfa69a02 P f14be40334fc4070a244ad995b5d88f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.627815 5095 ts_tablet_manager.cc:1905] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.629946 5094 ts_tablet_manager.cc:1905] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:09.630858 5095 ts_tablet_manager.cc:1918] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:34:09.630945 5095 log.cc:1196] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/ea2d490ea7f54f088f160eb2dfa69a02
I20260808 18:34:09.631266 5095 ts_tablet_manager.cc:1939] T ea2d490ea7f54f088f160eb2dfa69a02 P e9702100462a4b49a6e67bce98d8dddd: Deleting consensus metadata
I20260808 18:34:09.632228 4436 catalog_manager.cc:4841] TS e9702100462a4b49a6e67bce98d8dddd (127.27.251.194:43223): tablet ea2d490ea7f54f088f160eb2dfa69a02 (table test-table [id=41b778e22475474f9f3eeaf2bda3e2ea]) successfully deleted
I20260808 18:34:09.633034 5094 ts_tablet_manager.cc:1918] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 18:34:09.633105 5094 log.cc:1196] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/ea2d490ea7f54f088f160eb2dfa69a02
I20260808 18:34:09.633380 5094 ts_tablet_manager.cc:1939] T ea2d490ea7f54f088f160eb2dfa69a02 P 5188f6e007134d778eefc638755fd693: Deleting consensus metadata
I20260808 18:34:09.634042 4436 catalog_manager.cc:4841] TS 5188f6e007134d778eefc638755fd693 (127.27.251.195:39279): tablet ea2d490ea7f54f088f160eb2dfa69a02 (table test-table [id=41b778e22475474f9f3eeaf2bda3e2ea]) successfully deleted
W20260808 18:34:09.639874 4438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:45123 (error 108)
I20260808 18:34:09.640682 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 4686
I20260808 18:34:09.642172 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:09.640651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:45123 (local address 127.0.0.1:44386)
0808 18:34:09.641924 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:09.642012 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":84,"threads_started":1}
W20260808 18:34:09.643182 4438 catalog_manager.cc:4584] TS f14be40334fc4070a244ad995b5d88f0 (127.27.251.193:45123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea2d490ea7f54f088f160eb2dfa69a02: Network error: Client connection negotiation failed: client connection to 127.27.251.193:45123: connect: Connection refused (error 111)
I20260808 18:34:09.650784 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 4845
I20260808 18:34:09.659667 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 4418
2026-08-08T18:34:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3604 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[5150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[5150](info): set up 2 sockets
Aug 08 18:34:09 dist-test-slave-fd7l krb5kdc[5150](info): commencing operation
krb5kdc: starting...
W20260808 18:34:10.761121 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:10 dist-test-slave-fd7l krb5kdc[5150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:34:10.897589 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34263
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:34263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:11.013409 5165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:11.013676 5165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:11.013717 5165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:11.017859 5165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:11.017916 5165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:11.017936 5165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:11.017953 5165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:11.017968 5165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:11.023269 5165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:34263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:11.024312 5165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:11.025406 5165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:11.034157 5170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:11.034214 5165 server_base.cc:1034] running on GCE node
W20260808 18:34:11.034336 5171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:11.034183 5173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:11.034952 5165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:11.036053 5165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:11.037253 5165 hybrid_clock.cc:648] HybridClock initialized: now 1786214051037223 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T18:34:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1392 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:34:11 dist-test-slave-fd7l krb5kdc[5178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:34:11 dist-test-slave-fd7l krb5kdc[5178](info): set up 2 sockets
Aug 08 18:34:11 dist-test-slave-fd7l krb5kdc[5178](info): commencing operation
krb5kdc: starting...
W20260808 18:34:12.155377 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:34:12.281728 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33047
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:12.396317 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:12.396652 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:12.396698 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:12.400594 5191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:12.400651 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:12.400671 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:12.400687 5191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:12.400703 5191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:12.406014 5191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:33047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:12.407051 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:12.408078 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:12.417059 5199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:12.417073 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:12.417068 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:12.417164 5191 server_base.cc:1034] running on GCE node
I20260808 18:34:12.417838 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:12.418978 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:12.420171 5191 hybrid_clock.cc:648] HybridClock initialized: now 1786214052420136 us; error 44 us; skew 500 ppm
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:12.423125 5191 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:34:12.424000 5191 webserver.cc:460] Webserver started at http://127.27.251.254:39619/ using document root <none> and password file <none>
I20260808 18:34:12.424592 5191 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:12.424659 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:12.424834 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:12.426872 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "5b2e33d8402c460a9bf84a4940e4600f"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "e801196770de734a77b5fe6b3528ee45"
server_key_iv: "09148da9462e5bb557c5b91b1f294f15"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.427392 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "5b2e33d8402c460a9bf84a4940e4600f"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "e801196770de734a77b5fe6b3528ee45"
server_key_iv: "09148da9462e5bb557c5b91b1f294f15"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.431178 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 18:34:12.433534 5206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.434753 5191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:12.434882 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "5b2e33d8402c460a9bf84a4940e4600f"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "e801196770de734a77b5fe6b3528ee45"
server_key_iv: "09148da9462e5bb557c5b91b1f294f15"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.434995 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:12.449647 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:12.453022 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:12.453239 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:12.460731 5191 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:33047
I20260808 18:34:12.460788 5264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:33047 every 8 connection(s)
I20260808 18:34:12.461741 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:12.464627 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:12.468829 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5191
I20260808 18:34:12.468983 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:12.469285 28655 external_mini_cluster.cc:1428] Setting key c22b334d5af459605d9fd4411f02c46f
I20260808 18:34:12.470578 5265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Bootstrap starting.
I20260808 18:34:12.473373 5265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:12.474320 5265 log.cc:824] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:12.476408 5265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: No bootstrap required, opened a new log
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:12.480736 5265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:12.481021 5265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:12.481082 5265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2e33d8402c460a9bf84a4940e4600f, State: Initialized, Role: FOLLOWER
I20260808 18:34:12.481596 5265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:12.481729 5265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:12.481822 5265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:12.481922 5265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:12.482815 5265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:12.483174 5265 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2e33d8402c460a9bf84a4940e4600f; no voters:
I20260808 18:34:12.483589 5265 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:12.483673 5270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:12.483893 5270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 LEADER]: Becoming Leader. State: Replica: 5b2e33d8402c460a9bf84a4940e4600f, State: Running, Role: LEADER
I20260808 18:34:12.484216 5270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:12.484678 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:12.470772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.27.251.254:33047)
0808 18:34:12.471194 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:12.471215 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:12.471258 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:34:12.472132 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:12.472157 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:12.472589 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:12.472956 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:12.473760 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.475134 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.475992 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.476706 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.476877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:12.479331 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:12.479337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:12.479354 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:12.479388 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:12.481406 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.481869 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.481877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.481886 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.482049 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.482356 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.482360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.482362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.482770 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:12.482983 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:12.483288 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 18:34:12.483623 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260808 18:34:12.484984 5265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:12.487855 5272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2e33d8402c460a9bf84a4940e4600f. Latest consensus state: current_term: 1 leader_uuid: "5b2e33d8402c460a9bf84a4940e4600f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } } }
I20260808 18:34:12.487833 5271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b2e33d8402c460a9bf84a4940e4600f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } } }
I20260808 18:34:12.487993 5271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:12.488202 5272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: This master's current role is: LEADER
W20260808 18:34:12.489799 5285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:34:12.489887 5285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:34:12.489967 5287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:12.490681 5287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:12.495707 5287 catalog_manager.cc:1347] Generated new cluster ID: 4e3f5fc8c43f4bfe8d80e3c8616bf300
I20260808 18:34:12.495774 5287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:12.515614 5287 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:12.516683 5287 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:12.520574 5287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Generated new TSK 0
I20260808 18:34:12.521261 5287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:34:12.559648 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:12.681340 5295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:12.681604 5295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:12.681646 5295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:12.685772 5295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:12.685827 5295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:12.685912 5295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:12.691159 5295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:12.692209 5295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:12.693297 5295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:12.703091 5306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:12.703202 5302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:12.703094 5303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:12.703393 5295 server_base.cc:1034] running on GCE node
I20260808 18:34:12.703883 5295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:12.704644 5295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:12.705833 5295 hybrid_clock.cc:648] HybridClock initialized: now 1786214052705810 us; error 38 us; skew 500 ppm
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:12.708817 5295 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:34:12.709722 5295 webserver.cc:460] Webserver started at http://127.27.251.193:38057/ using document root <none> and password file <none>
I20260808 18:34:12.710314 5295 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:12.710405 5295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:12.710635 5295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:12.712745 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "fe939c884eb941cfb64e048aa4be1d34"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "7ba23f29a2c04921a58aedcf9b3b7771"
server_key_iv: "6ab5721763dca963a6de6b3c76b2bacc"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.713316 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "fe939c884eb941cfb64e048aa4be1d34"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "7ba23f29a2c04921a58aedcf9b3b7771"
server_key_iv: "6ab5721763dca963a6de6b3c76b2bacc"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.717561 5295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:34:12.719868 5313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.721306 5295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:12.721406 5295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "fe939c884eb941cfb64e048aa4be1d34"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "7ba23f29a2c04921a58aedcf9b3b7771"
server_key_iv: "6ab5721763dca963a6de6b3c76b2bacc"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.721493 5295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:12.737098 5295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:12.740296 5295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:12.740447 5295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:12.741073 5295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:12.741887 5295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:12.741933 5295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.742003 5295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:12.742033 5295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.751310 5295 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37197
I20260808 18:34:12.751330 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37197 every 8 connection(s)
I20260808 18:34:12.752231 5295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:12.757781 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5295
I20260808 18:34:12.757911 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:12.758210 28655 external_mini_cluster.cc:1428] Setting key 5188150388ea630b8fa0c7e5b1115d5b
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:12.767238 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:12.754046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:44383 (local address 127.27.251.254:33047)
0808 18:34:12.754219 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:12.754222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:12.754954 (+ 732us) server_negotiation.cc:407] Connection header received
0808 18:34:12.756250 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:12.756253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:12.756301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:12.756375 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:12.757532 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.758194 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.759632 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.759889 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.759970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:12.763152 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:12.763158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:12.763162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:12.763200 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:12.764875 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.765583 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.765586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.765587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.765666 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.766077 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.766080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.766081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.766259 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:12.766335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:12.767013 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 18:34:12.767116 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:34:12.768497 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:12.754361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:44383)
0808 18:34:12.754797 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:12.754840 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:12.756036 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:12.756574 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:12.756581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:12.756921 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:12.757389 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:12.757402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.758329 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.758332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:12.759508 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:12.759521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.760152 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.760158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:12.760386 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:12.761009 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:12.761036 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:12.762985 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:12.765023 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:12.765033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:12.765051 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:12.765475 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:12.765772 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:12.765775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:12.765778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:12.765977 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:12.766374 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:12.766384 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:12.766717 (+ 333us) client_negotiation.cc:769] Sending connection context
0808 18:34:12.766957 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 18:34:12.767349 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:34:12.769917 5434 heartbeater.cc:347] Connected to a master server at 127.27.251.254:33047
I20260808 18:34:12.770190 5434 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:12.770676 5434 heartbeater.cc:511] Master 127.27.251.254:33047 requested a full tablet report, sending...
I20260808 18:34:12.772228 5225 ts_manager.cc:194] Registered new tserver with Master: fe939c884eb941cfb64e048aa4be1d34 (127.27.251.193:37197)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:34:12.773617 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:44383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:34:12.788266 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:12.901966 5440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:12.902251 5440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:12.902336 5440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:12.906469 5440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:12.906530 5440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:12.906641 5440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:12.912068 5440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:12.913151 5440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:12.914256 5440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:12.924548 5448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:12.924628 5447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:12.924882 5440 server_base.cc:1034] running on GCE node
W20260808 18:34:12.924897 5450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:12.925311 5440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:12.926035 5440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:12.927248 5440 hybrid_clock.cc:648] HybridClock initialized: now 1786214052927230 us; error 37 us; skew 500 ppm
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:12.930092 5440 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:34:12.931010 5440 webserver.cc:460] Webserver started at http://127.27.251.194:43133/ using document root <none> and password file <none>
I20260808 18:34:12.931613 5440 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:12.931710 5440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:12.931941 5440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:12.934116 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "c670acf8d9314128a084be2601f8894d"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "151a90ec115170a1a29997bfe01cbd5c"
server_key_iv: "f0320908f04c8e96e9d7fa3f6e2c1fe5"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.934725 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "c670acf8d9314128a084be2601f8894d"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "151a90ec115170a1a29997bfe01cbd5c"
server_key_iv: "f0320908f04c8e96e9d7fa3f6e2c1fe5"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.939123 5440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:12.941636 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.942942 5440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:12.943074 5440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "c670acf8d9314128a084be2601f8894d"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "151a90ec115170a1a29997bfe01cbd5c"
server_key_iv: "f0320908f04c8e96e9d7fa3f6e2c1fe5"
server_key_version: "encryptionkey@0"
I20260808 18:34:12.943213 5440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:12.958776 5440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:12.961711 5440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:12.961877 5440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:12.962455 5440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:12.963274 5440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:12.963320 5440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.963361 5440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:12.963376 5440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:12.973393 5440 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:38215
I20260808 18:34:12.973423 5577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:38215 every 8 connection(s)
I20260808 18:34:12.974349 5440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:12.975037 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5440
I20260808 18:34:12.975176 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:12.975504 28655 external_mini_cluster.cc:1428] Setting key 3f30bac63b7b5a8b88b3bd95ca369776
Aug 08 18:34:12 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:34:12.990696 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:12.976612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:48501 (local address 127.27.251.254:33047)
0808 18:34:12.976748 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:12.976753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:12.977848 (+ 1095us) server_negotiation.cc:407] Connection header received
0808 18:34:12.979919 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:12.979923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:12.979978 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:12.980057 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:12.981376 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.981914 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.983187 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.983414 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.983549 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:12.986659 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:12.986662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:12.986664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:12.986695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:12.988381 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.988997 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.989000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.989002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.989081 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:12.989460 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:12.989463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:12.989464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:12.989677 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:12.989782 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:12.990363 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 18:34:12.990489 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:34:12.991511 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:12.977134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:48501)
0808 18:34:12.977654 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:12.977720 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:12.979679 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:12.980240 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:12.980247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:12.980692 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:12.981223 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:12.981238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.982050 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.982059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:12.983040 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:12.983050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:12.983574 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:12.983579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:12.983762 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:12.984426 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:12.984443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:12.986446 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:12.988582 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:12.988592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:12.988617 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:12.988879 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:12.989172 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:12.989174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:12.989175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:12.989325 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:12.989808 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:12.989822 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:12.990062 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 18:34:12.990298 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 18:34:12.990636 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260808 18:34:12.992839 5578 heartbeater.cc:347] Connected to a master server at 127.27.251.254:33047
I20260808 18:34:12.993105 5578 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:12.993625 5578 heartbeater.cc:511] Master 127.27.251.254:33047 requested a full tablet report, sending...
I20260808 18:34:12.994733 5225 ts_manager.cc:194] Registered new tserver with Master: c670acf8d9314128a084be2601f8894d (127.27.251.194:38215)
I20260808 18:34:12.995388 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:48501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:34:13.007393 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:13.123656 5584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:13.123968 5584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:13.124047 5584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:13.128451 5584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:13.128579 5584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:13.128705 5584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:13.134232 5584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:13.135373 5584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:13.136447 5584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:13.146757 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.146757 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.146968 5584 server_base.cc:1034] running on GCE node
W20260808 18:34:13.146786 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.147591 5584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:13.148259 5584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:13.149448 5584 hybrid_clock.cc:648] HybridClock initialized: now 1786214053149421 us; error 42 us; skew 500 ppm
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:13.152249 5584 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:34:13.153285 5584 webserver.cc:460] Webserver started at http://127.27.251.195:33609/ using document root <none> and password file <none>
I20260808 18:34:13.153875 5584 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:13.153970 5584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:13.154234 5584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:13.156330 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "f8cd603bddcd4669b4abd2f95b6afcfb"
format_stamp: "Formatted at 2026-08-08 18:34:13 on dist-test-slave-fd7l"
server_key: "86125ef4a7d25fc5261db2f50ff424f8"
server_key_iv: "70d39a1b241acde78bda7dae9b85bc81"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.156981 5584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "f8cd603bddcd4669b4abd2f95b6afcfb"
format_stamp: "Formatted at 2026-08-08 18:34:13 on dist-test-slave-fd7l"
server_key: "86125ef4a7d25fc5261db2f50ff424f8"
server_key_iv: "70d39a1b241acde78bda7dae9b85bc81"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.162118 5584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:34:13.164727 5601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.166054 5584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:13.166216 5584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "f8cd603bddcd4669b4abd2f95b6afcfb"
format_stamp: "Formatted at 2026-08-08 18:34:13 on dist-test-slave-fd7l"
server_key: "86125ef4a7d25fc5261db2f50ff424f8"
server_key_iv: "70d39a1b241acde78bda7dae9b85bc81"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.166352 5584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:13.178161 5584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:13.181090 5584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:13.181296 5584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:13.181898 5584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:13.182755 5584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:13.182826 5584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.182898 5584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:13.182940 5584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.192552 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:43565 every 8 connection(s)
I20260808 18:34:13.192553 5584 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:43565
I20260808 18:34:13.193564 5584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:13.195082 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5584
I20260808 18:34:13.195235 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:13.195597 28655 external_mini_cluster.cc:1428] Setting key ac3874de8df875ef0c3798df25de0ed2
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:13.207832 5268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.195565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:51431 (local address 127.27.251.254:33047)
0808 18:34:13.195735 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.195740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.196817 (+ 1077us) server_negotiation.cc:407] Connection header received
0808 18:34:13.198378 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.198382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.198437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.198517 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.199698 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.200237 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.201481 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.201729 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.201839 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.204337 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:13.204341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:13.204344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:13.204374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:13.205764 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:13.206280 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:13.206286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:13.206288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:13.206374 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:13.206718 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:13.206723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:13.206725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:13.206912 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:13.207025 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:13.207520 (+ 495us) server_negotiation.cc:299] Negotiation successful
0808 18:34:13.207656 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 18:34:13.208853 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.196080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:51431)
0808 18:34:13.196667 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.196727 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.198170 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.198700 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.198706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.199061 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.199551 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.199565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.200365 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.200368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.201350 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.201361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.201881 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.201885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.202067 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.202711 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.202728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.204176 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:13.205885 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:13.205891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:13.205905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:13.206172 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:13.206473 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:13.206476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:13.206477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:13.206617 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:13.207024 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:13.207030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:13.207283 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 18:34:13.207478 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 18:34:13.207799 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":152,"threads_started":1}
I20260808 18:34:13.209916 5726 heartbeater.cc:347] Connected to a master server at 127.27.251.254:33047
I20260808 18:34:13.210170 5726 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:13.210650 5726 heartbeater.cc:511] Master 127.27.251.254:33047 requested a full tablet report, sending...
I20260808 18:34:13.211758 5225 ts_manager.cc:194] Registered new tserver with Master: f8cd603bddcd4669b4abd2f95b6afcfb (127.27.251.195:43565)
I20260808 18:34:13.212292 5225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:51431
I20260808 18:34:13.221506 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
Entry for principal oryx/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab.
WARNING: no policy specified for oryx/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
Entry for principal oryx/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab.
WARNING: no policy specified for oryx/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
Entry for principal oryx/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab.
I20260808 18:34:13.305075 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5295
I20260808 18:34:13.313848 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5440
I20260808 18:34:13.321170 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5584
I20260808 18:34:13.330832 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5191
I20260808 18:34:13.339164 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33047
--webserver_interface=127.27.251.254
--webserver_port=39619
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:13.451498 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:13.451803 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:13.451877 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:13.455869 5736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:13.455962 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:13.456001 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:13.456053 5736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:13.456089 5736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:13.461364 5736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:33047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=39619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:13.462533 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:13.463913 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:13.475069 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.475111 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.475239 5736 server_base.cc:1034] running on GCE node
W20260808 18:34:13.475070 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.476117 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:13.477293 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:13.478500 5736 hybrid_clock.cc:648] HybridClock initialized: now 1786214053478473 us; error 45 us; skew 500 ppm
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:13.481523 5736 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:34:13.482404 5736 webserver.cc:460] Webserver started at http://127.27.251.254:39619/ using document root <none> and password file <none>
I20260808 18:34:13.482987 5736 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:13.483104 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:13.487039 5736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:34:13.489131 5752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.490398 5736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:13.490572 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "5b2e33d8402c460a9bf84a4940e4600f"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "e801196770de734a77b5fe6b3528ee45"
server_key_iv: "09148da9462e5bb557c5b91b1f294f15"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.491118 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:13.503021 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:13.506265 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:13.506479 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:13.513455 5736 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:33047
I20260808 18:34:13.513573 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:33047 every 8 connection(s)
I20260808 18:34:13.514622 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:13.515725 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5736
I20260808 18:34:13.516242 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:37197
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:33047
--webserver_port=38057
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:34:13.519904 5812 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:34:13.521636 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Bootstrap starting.
I20260808 18:34:13.539322 5812 log.cc:824] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:13.546026 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:34:13.546418 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Bootstrap complete.
I20260808 18:34:13.550817 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:13.551287 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b2e33d8402c460a9bf84a4940e4600f, State: Initialized, Role: FOLLOWER
I20260808 18:34:13.551785 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:13.551898 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:13.551991 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:13.552093 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:34:13.554342 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:13.554692 5812 leader_election.cc:302] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b2e33d8402c460a9bf84a4940e4600f; no voters:
I20260808 18:34:13.555052 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:34:13.555157 5816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:34:13.555344 5816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [term 2 LEADER]: Becoming Leader. State: Replica: 5b2e33d8402c460a9bf84a4940e4600f, State: Running, Role: LEADER
I20260808 18:34:13.555631 5816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } }
I20260808 18:34:13.556311 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:13.556620 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b2e33d8402c460a9bf84a4940e4600f. Latest consensus state: current_term: 2 leader_uuid: "5b2e33d8402c460a9bf84a4940e4600f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } } }
I20260808 18:34:13.556739 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:13.557016 5817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b2e33d8402c460a9bf84a4940e4600f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b2e33d8402c460a9bf84a4940e4600f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 33047 } } }
I20260808 18:34:13.557109 5820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:13.557111 5817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:13.560475 5820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:13.561378 5820 catalog_manager.cc:1259] Loaded cluster ID: 4e3f5fc8c43f4bfe8d80e3c8616bf300
I20260808 18:34:13.561447 5820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:13.563812 5820 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:13.564466 5820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b2e33d8402c460a9bf84a4940e4600f: Loaded TSK: 0
I20260808 18:34:13.565706 5820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:34:13.634721 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:13.635030 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:13.635113 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:13.639111 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:13.639204 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:13.639294 5814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:13.644577 5814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.193.keytab
--principal=oryx/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:37197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=38057
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:13.645603 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:13.646643 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:13.656490 5843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.656517 5844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.656490 5846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.657058 5814 server_base.cc:1034] running on GCE node
I20260808 18:34:13.657541 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:13.658360 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:13.659572 5814 hybrid_clock.cc:648] HybridClock initialized: now 1786214053659547 us; error 46 us; skew 500 ppm
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214053, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:13.663129 5814 init.cc:377] Logged in from keytab as oryx/127.27.251.193@KRBTEST.COM (short username oryx)
I20260808 18:34:13.664268 5814 webserver.cc:460] Webserver started at http://127.27.251.193:38057/ using document root <none> and password file <none>
I20260808 18:34:13.665005 5814 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:13.665130 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:13.670328 5814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:34:13.672984 5854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.674594 5814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:34:13.674719 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "fe939c884eb941cfb64e048aa4be1d34"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "7ba23f29a2c04921a58aedcf9b3b7771"
server_key_iv: "6ab5721763dca963a6de6b3c76b2bacc"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.675434 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:13.699049 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:13.703050 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:13.703298 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:13.704088 5814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:13.705164 5814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:13.705231 5814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.705283 5814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:13.705313 5814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.715927 5814 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37197
I20260808 18:34:13.716013 5973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37197 every 8 connection(s)
I20260808 18:34:13.717216 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:13.724220 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5814
I20260808 18:34:13.724785 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:38215
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:33047
--webserver_port=43133
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:13.731918 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.719423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56453 (local address 127.27.251.254:33047)
0808 18:34:13.719859 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.719874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.720170 (+ 296us) server_negotiation.cc:407] Connection header received
0808 18:34:13.721765 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.721776 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.722062 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.722361 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.723659 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.725151 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.726368 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.727014 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.727148 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.731139 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:56453: BlockingRecv error: recv got EOF from 127.27.251.193:56453 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":132,"threads_started":1}
I20260808 18:34:13.731927 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.719501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:56453)
0808 18:34:13.719989 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.720041 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.721409 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.722491 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.722498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.722982 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.723477 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.723492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.725295 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.725298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.726208 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.726222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.727201 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.727208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.727440 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.728116 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.728143 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.730924 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":157,"threads_started":1}
W20260808 18:34:13.732409 5974 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:33047 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.251.254:33047: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:13.738034 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.732921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:60427)
0808 18:34:13.733049 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.733063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.733162 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.733394 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.733396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.733563 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.733653 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.733658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.734370 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.734375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.734927 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.734935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.735389 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.735392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.735494 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.735977 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.735990 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.737901 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 18:34:13.738256 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.733049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:60427 (local address 127.27.251.254:33047)
0808 18:34:13.733183 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.733190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.733206 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:13.733248 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.733255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.733294 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.733379 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.733769 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.734261 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.735047 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.735265 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.735371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.738025 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:60427: BlockingRecv error: recv got EOF from 127.27.251.193:60427 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:13.743669 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.738521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:45893)
0808 18:34:13.738677 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.738689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.738754 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.738971 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.738974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.739133 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.739220 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.739226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.739873 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.739876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.740789 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.740799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.741209 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.741212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.741294 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.741716 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.741730 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.743537 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260808 18:34:13.743762 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.738637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:45893 (local address 127.27.251.254:33047)
0808 18:34:13.738743 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.738746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.738758 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:34:13.738832 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.738834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.738867 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.738929 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.739325 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.739777 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.740896 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.741105 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.741182 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.743654 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:45893: BlockingRecv error: recv got EOF from 127.27.251.193:45893 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 18:34:13.743872 5974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:34:13.842752 5979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:13.842993 5979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:13.843032 5979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:13.846962 5979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:13.847014 5979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:13.847069 5979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:13.852059 5979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.194.keytab
--principal=oryx/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:38215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=43133
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:13.853189 5979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:13.854120 5979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:13.864614 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.864682 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:13.864761 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:13.864815 5979 server_base.cc:1034] running on GCE node
I20260808 18:34:13.865423 5979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:13.866108 5979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:13.867295 5979 hybrid_clock.cc:648] HybridClock initialized: now 1786214053867272 us; error 29 us; skew 500 ppm
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214053, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:13.870429 5979 init.cc:377] Logged in from keytab as oryx/127.27.251.194@KRBTEST.COM (short username oryx)
I20260808 18:34:13.871351 5979 webserver.cc:460] Webserver started at http://127.27.251.194:43133/ using document root <none> and password file <none>
I20260808 18:34:13.871913 5979 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:13.872015 5979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:13.875789 5979 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:34:13.877810 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.879011 5979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:13.879164 5979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "c670acf8d9314128a084be2601f8894d"
format_stamp: "Formatted at 2026-08-08 18:34:12 on dist-test-slave-fd7l"
server_key: "151a90ec115170a1a29997bfe01cbd5c"
server_key_iv: "f0320908f04c8e96e9d7fa3f6e2c1fe5"
server_key_version: "encryptionkey@0"
I20260808 18:34:13.879704 5979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:13.906365 5979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:13.909492 5979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:13.909713 5979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:13.910368 5979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:13.911232 5979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:13.911279 5979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.911350 5979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:13.911392 5979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:13.920874 5979 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:38215
I20260808 18:34:13.920899 6114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:38215 every 8 connection(s)
I20260808 18:34:13.921893 5979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:13.922933 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 5979
I20260808 18:34:13.923458 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:43565
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:33047
--webserver_port=33609
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:13.937049 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.924024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:57765 (local address 127.27.251.254:33047)
0808 18:34:13.924189 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.924193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.925323 (+ 1130us) server_negotiation.cc:407] Connection header received
0808 18:34:13.927549 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.927554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.927629 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.927734 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.929698 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.930729 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.932743 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.933221 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.933364 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.936859 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:57765: BlockingRecv error: recv got EOF from 127.27.251.194:57765 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:34:13.937505 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.924582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:57765)
0808 18:34:13.925130 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.925177 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.927294 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.928012 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.928022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.928667 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.929490 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.929509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.930897 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.930901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.932587 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.932601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.933429 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.933433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.933581 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.934226 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.934244 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.936649 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
W20260808 18:34:13.938133 6115 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:33047 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.251.254:33047: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:13.944178 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.938568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:52657)
0808 18:34:13.938767 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.938790 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.938855 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.939086 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.939088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.939225 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.939311 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.939316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.940419 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.940422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.941009 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.941018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.941637 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.941640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.941707 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.942215 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.942227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.944079 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260808 18:34:13.944650 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.938714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:52657 (local address 127.27.251.254:33047)
0808 18:34:13.938854 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.938859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.938876 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:34:13.938919 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.938922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.938973 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.939064 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.939415 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.940316 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.941119 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.941518 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.941621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.944251 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:52657: BlockingRecv error: recv got EOF from 127.27.251.194:52657 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:13 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:13.950239 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:13.944772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:60115)
0808 18:34:13.944930 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:13.944944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:13.945013 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:13.945291 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:13.945293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:13.945412 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:13.945477 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.945482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.946550 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.946552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.947042 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:13.947048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.947644 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.947646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:13.947700 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.948188 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:13.948200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:13.950157 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260808 18:34:13.950501 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:13.944902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:60115 (local address 127.27.251.254:33047)
0808 18:34:13.945071 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:13.945078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:13.945094 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:13.945132 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:13.945135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:13.945177 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:13.945266 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:13.945587 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.946436 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.947152 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:13.947534 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:13.947664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:13.950324 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:60115: BlockingRecv error: recv got EOF from 127.27.251.194:60115 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 18:34:13.950515 6115 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:34:14.054896 6119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:14.055150 6119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:14.055194 6119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:14.059661 6119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:14.059718 6119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:14.059777 6119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:14.065289 6119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/krb5kdc/oryx_127.27.251.195.keytab
--principal=oryx/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:43565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=33609
--tserver_master_addrs=127.27.251.254:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:14.066376 6119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:14.067442 6119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:14.077382 6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:14.077370 6125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:14.077373 6124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:14.077455 6119 server_base.cc:1034] running on GCE node
I20260808 18:34:14.078120 6119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:14.078840 6119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:14.080045 6119 hybrid_clock.cc:648] HybridClock initialized: now 1786214054080018 us; error 30 us; skew 500 ppm
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214054, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:14.083109 6119 init.cc:377] Logged in from keytab as oryx/127.27.251.195@KRBTEST.COM (short username oryx)
I20260808 18:34:14.083931 6119 webserver.cc:460] Webserver started at http://127.27.251.195:33609/ using document root <none> and password file <none>
I20260808 18:34:14.084578 6119 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:14.084661 6119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:14.088368 6119 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:34:14.090469 6134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:14.091727 6119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:14.091852 6119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "f8cd603bddcd4669b4abd2f95b6afcfb"
format_stamp: "Formatted at 2026-08-08 18:34:13 on dist-test-slave-fd7l"
server_key: "86125ef4a7d25fc5261db2f50ff424f8"
server_key_iv: "70d39a1b241acde78bda7dae9b85bc81"
server_key_version: "encryptionkey@0"
I20260808 18:34:14.092442 6119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:14.130195 6119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:14.133438 6119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:14.133646 6119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:14.134321 6119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:14.135260 6119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:14.135308 6119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:14.135377 6119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:14.135421 6119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:14.144966 6119 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:43565
I20260808 18:34:14.145006 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:43565 every 8 connection(s)
I20260808 18:34:14.145993 6119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:14.153151 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6119
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:14.158221 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:14.147901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:39767 (local address 127.27.251.254:33047)
0808 18:34:14.148082 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.148087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.148982 (+ 895us) server_negotiation.cc:407] Connection header received
0808 18:34:14.150327 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.150330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.150380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.150459 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.151665 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.152280 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.153535 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.153800 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.153914 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.158091 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:39767: BlockingRecv error: recv got EOF from 127.27.251.195:39767 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 18:34:14.159026 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.148316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:39767)
0808 18:34:14.148820 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:14.148865 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:14.150102 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:14.150640 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:14.150646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:14.151003 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:14.151496 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.151509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.152427 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.152430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.153408 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.153419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.154001 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.154007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.154265 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.154987 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:14.155013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:14.157982 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":179,"threads_started":1}
W20260808 18:34:14.159521 6255 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:33047 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.251.254:33047: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:14.166280 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.159910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:56061)
0808 18:34:14.160027 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:14.160038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:14.160148 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:14.160413 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:14.160416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:14.160625 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:14.160716 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.160721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.161735 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.161738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.162623 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.162644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.163384 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.163388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.163489 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.163989 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:14.164006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:14.166116 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:34:14.166353 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.154334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.27.251.254:33047)
0808 18:34:14.154656 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.154660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.154675 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:14.154713 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.154715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.154757 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.154890 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.155690 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.156412 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.157675 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.157988 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.158130 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.160732 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:14.160738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:14.160753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:14.160779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:14.163772 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:14.164436 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:14.164442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:14.164449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:14.164663 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:14.164971 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:14.164975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:14.164978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:14.165446 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:14.165659 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:14.165933 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 18:34:14.166214 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
W20260808 18:34:14.166496 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:14.160019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:56061 (local address 127.27.251.254:33047)
0808 18:34:14.160141 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.160145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.160159 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:14.160263 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.160266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.160305 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.160398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.160830 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.161585 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.162783 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.163130 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.163215 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.166330 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:56061: BlockingRecv error: recv got EOF from 127.27.251.195:56061 (error 108)
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":49}
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:14.172014 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.167166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:39531)
0808 18:34:14.167318 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:14.167331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:14.167403 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:14.167604 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:14.167607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:14.167761 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:14.167848 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.167853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.168606 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.168613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.169382 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.169392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.169791 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.169794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.169850 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.170306 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:14.170319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:14.171916 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 18:34:14.172178 5976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:14.167270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:39531 (local address 127.27.251.254:33047)
0808 18:34:14.167416 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.167420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.167432 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:34:14.167480 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.167481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.167513 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.167575 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.167954 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.168439 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.169500 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.169701 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.169780 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.172024 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:39531: BlockingRecv error: recv got EOF from 127.27.251.195:39531 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 18:34:14.172250 6255 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:14.750260 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.744685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:37833)
0808 18:34:14.745013 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:14.745037 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:14.745144 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:14.745387 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:14.745389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:14.745573 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:14.745687 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.745692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.746432 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.746435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.747017 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.747025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.747448 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.747451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.747547 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.748127 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:14.748139 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:14.750085 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
W20260808 18:34:14.750376 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:14.744702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:37833 (local address 127.27.251.254:33047)
0808 18:34:14.744977 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.744981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.745118 (+ 137us) server_negotiation.cc:407] Connection header received
0808 18:34:14.745225 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.745227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.745274 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.745343 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.745792 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.746326 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.747130 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.747351 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.747426 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.750241 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:37833: BlockingRecv error: recv got EOF from 127.27.251.193:37833 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:14 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:14.957367 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:14.951155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:37627)
0808 18:34:14.951496 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:14.951511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:14.951676 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:14.951960 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:14.951963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:14.952187 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:14.952281 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.952287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.953140 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.953146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.953746 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:14.953754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.954247 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.954251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:14.954339 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.954876 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:14.954888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:14.957176 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
W20260808 18:34:14.957459 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:14.951292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:37627 (local address 127.27.251.254:33047)
0808 18:34:14.951456 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:14.951461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:14.951594 (+ 133us) server_negotiation.cc:407] Connection header received
0808 18:34:14.951797 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:14.951800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:14.951846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:14.951935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:14.952402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.953022 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.953871 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:14.954117 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:14.954231 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:14.957307 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:37627: BlockingRecv error: recv got EOF from 127.27.251.194:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:15.178872 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:15.172895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:34497)
0808 18:34:15.173243 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:15.173258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:15.173406 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:15.173731 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:15.173734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:15.173957 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:15.174058 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.174064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.174893 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.174898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.175495 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.175504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.175965 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.175969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.176054 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.176740 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:15.176753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:15.178745 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
W20260808 18:34:15.179006 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:15.173015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:34497 (local address 127.27.251.254:33047)
0808 18:34:15.173196 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:15.173201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:15.173348 (+ 147us) server_negotiation.cc:407] Connection header received
0808 18:34:15.173561 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:15.173564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:15.173620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:15.173707 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:15.174220 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.174766 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.175620 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.175849 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.175967 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.178884 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:34497: BlockingRecv error: recv got EOF from 127.27.251.195:34497 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:15.757597 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:15.751428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:33793 (local address 127.27.251.254:33047)
0808 18:34:15.751941 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:15.751947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:15.751965 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:34:15.752015 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:15.752017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:15.752079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:15.752187 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:15.752794 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.753408 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.754242 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.754485 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.754603 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.757483 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:33793: BlockingRecv error: recv got EOF from 127.27.251.193:33793 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":113,"threads_started":1}
I20260808 18:34:15.757594 6264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:15.751355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:33793)
0808 18:34:15.751653 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:15.751667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:15.751763 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:15.752222 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:15.752225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:15.752467 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:15.752661 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.752675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.753535 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.753539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.754113 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.754122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.754609 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.754612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.754700 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.755265 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:15.755278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:15.757361 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:15 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:15.964202 6266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:15.958461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:47593)
0808 18:34:15.958746 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:15.958759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:15.958870 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:15.959138 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:15.959141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:15.959328 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:15.959432 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.959438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.960190 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.960193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.960776 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:15.960783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.961270 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.961274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:15.961388 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.962050 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:15.962066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:15.964080 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
W20260808 18:34:15.964334 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:15.958557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:47593 (local address 127.27.251.254:33047)
0808 18:34:15.958749 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:15.958753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:15.958837 (+ 84us) server_negotiation.cc:407] Connection header received
0808 18:34:15.958954 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:15.958956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:15.959004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:15.959103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:15.959568 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.960087 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.960898 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:15.961129 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:15.961221 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:15.964214 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:47593: BlockingRecv error: recv got EOF from 127.27.251.194:47593 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:16.190064 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:16.179816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:57537)
0808 18:34:16.180112 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:16.180125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:16.180236 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:16.180599 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:16.180601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:16.180836 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:16.180942 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.180948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.181828 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.181836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.182426 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.182436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.183068 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.183071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.183182 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.187773 (+ 4591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:16.187797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:16.189909 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
W20260808 18:34:16.190264 6265 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:16.179924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:57537 (local address 127.27.251.254:33047)
0808 18:34:16.180104 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:16.180109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:16.180181 (+ 72us) server_negotiation.cc:407] Connection header received
0808 18:34:16.180322 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:16.180324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:16.180413 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:16.180528 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:16.181068 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.181676 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.182534 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.182760 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.182861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.190090 (+ 7229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:57537: BlockingRecv error: recv got EOF from 127.27.251.195:57537 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:16.764125 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:16.758367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:34521)
0808 18:34:16.758701 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:16.758717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:16.758835 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:16.759079 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:16.759081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:16.759291 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:16.759425 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.759436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.760499 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.760528 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.761144 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.761154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.761576 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.761578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.761639 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.762133 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:16.762160 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:16.764007 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
W20260808 18:34:16.764307 6270 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:16.758452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:34521 (local address 127.27.251.254:33047)
0808 18:34:16.758759 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:16.758762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:16.758777 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:16.758921 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:16.758923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:16.758970 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:16.759049 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:16.759556 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.760381 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.761255 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.761473 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.761555 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.764171 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:34521: BlockingRecv error: recv got EOF from 127.27.251.193:34521 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":133,"threads_started":1}
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:16 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:16.970870 6270 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:16.965353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:37199 (local address 127.27.251.254:33047)
0808 18:34:16.965543 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:16.965546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:16.965561 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:16.965700 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:16.965702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:16.965749 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:16.965818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:16.966299 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.966893 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.967703 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.967935 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.968020 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.970769 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:37199: BlockingRecv error: recv got EOF from 127.27.251.194:37199 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 18:34:16.970909 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:16.965216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:37199)
0808 18:34:16.965492 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:16.965506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:16.965608 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:16.965857 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:16.965860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:16.966045 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:16.966160 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.966165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.967015 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.967017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.967576 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:16.967584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:16.968042 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:16.968045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:16.968107 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:16.968696 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:16.968709 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:16.970555 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:17.197053 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:17.191012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:40563)
0808 18:34:17.191307 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:17.191329 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:17.191473 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:17.191829 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:17.191832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:17.192057 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:17.192154 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.192161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.193029 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.193033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.193619 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.193627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.194099 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.194102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.194168 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.194737 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:17.194756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:17.196863 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20260808 18:34:17.197111 6270 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:17.191071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:40563 (local address 127.27.251.254:33047)
0808 18:34:17.191282 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:17.191287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:17.191408 (+ 121us) server_negotiation.cc:407] Connection header received
0808 18:34:17.191650 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:17.191653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:17.191714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:17.191806 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:17.192275 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.192916 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.193771 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.193999 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.194079 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.196988 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:40563: BlockingRecv error: recv got EOF from 127.27.251.195:40563 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:17.771605 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:17.764985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:59769)
0808 18:34:17.765351 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:17.765369 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:17.765504 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:17.765840 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:17.765845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:17.766112 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:17.766233 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.766240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.767358 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.767360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.767914 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.767923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.768591 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.768601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.768694 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.769260 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:17.769274 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:17.771363 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":90,"threads_started":1}
W20260808 18:34:17.771628 6275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:17.765115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:59769 (local address 127.27.251.254:33047)
0808 18:34:17.765396 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:17.765399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:17.765439 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:34:17.765622 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:17.765628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:17.765698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:17.765776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:17.766373 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.767243 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.768034 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.768329 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.768441 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.771501 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:59769: BlockingRecv error: recv got EOF from 127.27.251.193:59769 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:17 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:17.977948 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:17.972019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:54093)
0808 18:34:17.972311 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:17.972325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:17.972438 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:17.972734 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:17.972736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:17.972923 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:17.973054 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.973060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.973972 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.973975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.974563 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:17.974571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.975045 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.975048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:17.975116 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.975708 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:17.975721 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:17.977771 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
W20260808 18:34:17.978032 6275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:17.972092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:54093 (local address 127.27.251.254:33047)
0808 18:34:17.972269 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:17.972277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:17.972430 (+ 153us) server_negotiation.cc:407] Connection header received
0808 18:34:17.972558 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:17.972561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:17.972612 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:17.972688 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:17.973229 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.973852 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.974684 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:17.974907 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:17.974995 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:17.977915 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:54093: BlockingRecv error: recv got EOF from 127.27.251.194:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:18.203575 6277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:18.197974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:53351)
0808 18:34:18.198302 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:18.198325 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:18.198473 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:18.198786 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:18.198789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:18.199006 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:18.199131 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.199138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.199913 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.199919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.200482 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.200490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.201000 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.201003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.201099 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.201608 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:18.201620 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:18.203452 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
W20260808 18:34:18.203722 6275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:18.198095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:53351 (local address 127.27.251.254:33047)
0808 18:34:18.198263 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:18.198268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:18.198411 (+ 143us) server_negotiation.cc:407] Connection header received
0808 18:34:18.198617 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:18.198620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:18.198670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:18.198763 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:18.199259 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.199799 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.200656 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.200879 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.200992 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.203601 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:53351: BlockingRecv error: recv got EOF from 127.27.251.195:53351 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:18.778472 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:18.772704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:41437)
0808 18:34:18.772989 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:18.773003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:18.773113 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:18.773369 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:18.773371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:18.773616 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:18.773725 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.773731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.774502 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.774505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.775136 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.775146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.775563 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.775565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.775634 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.776154 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:18.776167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:18.778287 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260808 18:34:18.778517 6279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:18.772738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:41437 (local address 127.27.251.254:33047)
0808 18:34:18.772996 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:18.773000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:18.773078 (+ 78us) server_negotiation.cc:407] Connection header received
0808 18:34:18.773198 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:18.773200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:18.773245 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:18.773328 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:18.773831 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.774379 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.775249 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.775463 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.775538 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.778397 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:41437: BlockingRecv error: recv got EOF from 127.27.251.193:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:18 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:18.984761 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:18.978855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:48003)
0808 18:34:18.979156 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:18.979169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:18.979264 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:18.979582 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:18.979584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:18.979777 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:18.979876 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.979881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.980836 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.980838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.981434 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:18.981442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.981916 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.981919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:18.981984 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.982491 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:18.982504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:18.984642 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
W20260808 18:34:18.984889 6279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:18.978991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:48003 (local address 127.27.251.254:33047)
0808 18:34:18.979167 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:18.979171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:18.979290 (+ 119us) server_negotiation.cc:407] Connection header received
0808 18:34:18.979342 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:18.979346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:18.979407 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:18.979522 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:18.980018 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.980726 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.981580 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:18.981800 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:18.981880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:18.984756 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:48003: BlockingRecv error: recv got EOF from 127.27.251.194:48003 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:19.210058 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:19.204639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:51361)
0808 18:34:19.204942 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:19.204955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:19.205051 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:19.205296 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:19.205298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:19.205496 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:19.205618 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.205624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.206349 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.206351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.206958 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.206967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.207428 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.207432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.207563 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.208104 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:19.208116 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:19.209940 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
W20260808 18:34:19.210217 6279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:19.204720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:51361 (local address 127.27.251.254:33047)
0808 18:34:19.204862 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:19.204866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:19.205024 (+ 158us) server_negotiation.cc:407] Connection header received
0808 18:34:19.205134 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:19.205136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:19.205182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:19.205251 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:19.205724 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.206252 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.207064 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.207296 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.207375 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.210081 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:51361: BlockingRecv error: recv got EOF from 127.27.251.195:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:19.784929 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:19.779297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:56337)
0808 18:34:19.779587 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:19.779600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:19.779692 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:19.779942 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:19.779944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:19.780143 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:19.780289 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.780295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.781136 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.781138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.781819 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.781829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.782302 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.782304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.782364 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.782875 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:19.782895 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:19.784812 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
W20260808 18:34:19.785243 6283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:19.779379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56337 (local address 127.27.251.254:33047)
0808 18:34:19.779623 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:19.779627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:19.779645 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:34:19.779782 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:19.779784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:19.779830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:19.779907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:19.780393 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.781040 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.781923 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.782205 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.782285 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.784952 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:56337: BlockingRecv error: recv got EOF from 127.27.251.193:56337 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":60,"threads_started":1}
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:19 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:19.991425 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:19.985674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:42931)
0808 18:34:19.986020 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:19.986035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:19.986167 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:19.986421 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:19.986423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:19.986623 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:19.986728 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.986734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.987487 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.987489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.988050 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:19.988058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.988544 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.988549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:19.988655 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.989267 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:19.989289 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:19.991306 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
W20260808 18:34:19.991631 6283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:19.985764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42931 (local address 127.27.251.254:33047)
0808 18:34:19.985932 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:19.985937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:19.986118 (+ 181us) server_negotiation.cc:407] Connection header received
0808 18:34:19.986257 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:19.986260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:19.986307 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:19.986378 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:19.986845 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.987388 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.988154 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:19.988373 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:19.988491 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:19.991466 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:42931: BlockingRecv error: recv got EOF from 127.27.251.194:42931 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:20.217262 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:20.210973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:35949)
0808 18:34:20.211362 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:20.211382 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:20.211603 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:20.211931 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:20.211934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:20.212196 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:20.212306 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.212313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.213260 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.213265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.213900 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.213909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.214451 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.214455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.214561 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.215096 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:20.215109 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:20.217141 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":96,"threads_started":1}
W20260808 18:34:20.217486 6283 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:20.211152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:35949 (local address 127.27.251.254:33047)
0808 18:34:20.211362 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:20.211368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:20.211526 (+ 158us) server_negotiation.cc:407] Connection header received
0808 18:34:20.211737 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:20.211740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:20.211801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:20.211896 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:20.212452 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.213129 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.214041 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.214306 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.214433 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.217334 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:35949: BlockingRecv error: recv got EOF from 127.27.251.195:35949 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 18:34:20.795046 6288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:20.786052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:60889 (local address 127.27.251.254:33047)
0808 18:34:20.788356 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:20.788361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:20.788378 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:34:20.788426 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:20.788430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:20.788491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:20.788615 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:20.789330 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.790241 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.791287 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.791642 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.791727 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.794927 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:60889: BlockingRecv error: recv got EOF from 127.27.251.193:60889 (error 108)
Metrics: {"server-negotiator.queue_time_us":2201,"thread_start_us":93,"threads_started":1}
I20260808 18:34:20.795116 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:20.785766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:60889)
0808 18:34:20.786154 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:20.786167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:20.786260 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:20.788668 (+ 2408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:20.788671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:20.788979 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:20.789153 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.789162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.790363 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.790367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.791147 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.791160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.791775 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.791782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.791901 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.792443 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:20.792462 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:20.794792 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":99,"threads_started":1}
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:20 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:20.998757 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:20.992276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:49571)
0808 18:34:20.992695 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:20.992708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:20.992851 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:20.993199 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:20.993206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:20.993457 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:20.993609 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.993617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.994672 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.994675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.995482 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:20.995494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.996078 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.996081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:20.996153 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.996687 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:20.996706 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:20.998644 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":86,"threads_started":1}
W20260808 18:34:20.998903 6288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:20.992437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:49571 (local address 127.27.251.254:33047)
0808 18:34:20.992654 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:20.992659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:20.992794 (+ 135us) server_negotiation.cc:407] Connection header received
0808 18:34:20.992964 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:20.992966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:20.993044 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:20.993138 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:20.993775 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.994519 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.995662 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:20.995962 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:20.996043 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:20.998782 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:49571: BlockingRecv error: recv got EOF from 127.27.251.194:49571 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 08 18:34:21 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:21 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:21.228176 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:21.218011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:55321)
0808 18:34:21.218362 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:21.218375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:21.218474 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:21.218715 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:21.218717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:21.218899 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:21.219001 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:21.219007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.219772 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.219774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:21.220318 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:21.220325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.220855 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.220858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:21.220959 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:21.225788 (+ 4829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:21.225838 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:21.228053 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
W20260808 18:34:21.228319 6288 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:21.218132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:55321 (local address 127.27.251.254:33047)
0808 18:34:21.218304 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:21.218308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:21.218450 (+ 142us) server_negotiation.cc:407] Connection header received
0808 18:34:21.218559 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:21.218561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:21.218610 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:21.218679 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:21.219108 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.219672 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.220422 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.220751 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.220835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:21.228194 (+ 7359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:55321: BlockingRecv error: recv got EOF from 127.27.251.195:55321 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 08 18:34:21 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:21 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:21.802002 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:21.796144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:56447)
0808 18:34:21.796452 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:21.796472 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:21.796602 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:21.796883 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:21.796887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:21.797103 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:21.797214 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:21.797220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.797984 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.797987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:21.798557 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:21.798565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.798996 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.798998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:21.799067 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:21.799579 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:21.799591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:21.801825 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
W20260808 18:34:21.802136 6293 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:21.796266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56447 (local address 127.27.251.254:33047)
0808 18:34:21.796586 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:21.796590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:21.796605 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:21.796688 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:21.796690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:21.796738 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:21.796822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:21.797330 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.797871 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.798665 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:21.798889 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:21.798965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:21.801989 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:56447: BlockingRecv error: recv got EOF from 127.27.251.193:56447 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
Aug 08 18:34:21 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:22 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:22.005874 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:21.999668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:36985)
0808 18:34:21.999953 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:21.999967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:22.000102 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:22.000366 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:22.000368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:22.000602 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:22.000711 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.000716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.001630 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.001632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.002263 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.002274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.002789 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.002795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.002878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.003526 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:22.003543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:22.005692 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
W20260808 18:34:22.005936 6293 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:21.999726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36985 (local address 127.27.251.254:33047)
0808 18:34:21.999884 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:21.999888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:22.000053 (+ 165us) server_negotiation.cc:407] Connection header received
0808 18:34:22.000195 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:22.000198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:22.000248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:22.000323 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:22.000830 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.001522 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.002384 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.002644 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.002740 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.005824 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:36985: BlockingRecv error: recv got EOF from 127.27.251.194:36985 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 18:34:22 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:22 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:22.234706 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:22.229153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:56803)
0808 18:34:22.229444 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:22.229457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:22.229559 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:22.229788 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:22.229790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:22.229975 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:22.230078 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.230083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.230843 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.230845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.231393 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.231401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.231850 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.231852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.231916 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.232534 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:22.232548 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:22.234583 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
W20260808 18:34:22.234890 6293 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:22.229199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:56803 (local address 127.27.251.254:33047)
0808 18:34:22.229344 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:22.229348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:22.229533 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:34:22.229628 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:22.229630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:22.229678 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:22.229747 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:22.230184 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.230740 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.231520 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.231745 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.231828 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.234726 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:56803: BlockingRecv error: recv got EOF from 127.27.251.195:56803 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 18:34:22 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:22 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:22.809131 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:22.803163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:47731)
0808 18:34:22.803473 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:22.803487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:22.803599 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:22.803886 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:22.803888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:22.804095 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:22.804201 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.804207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.805146 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.805149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.805879 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:22.805894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.806416 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.806419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:22.806485 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.807004 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:22.807016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:22.809003 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
W20260808 18:34:22.809288 6297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:22.803269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:47731 (local address 127.27.251.254:33047)
0808 18:34:22.803549 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:22.803553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:22.803569 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:22.803709 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:22.803711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:22.803760 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:22.803844 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:22.804312 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.805034 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.806001 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:22.806311 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:22.806393 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:22.809148 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:47731: BlockingRecv error: recv got EOF from 127.27.251.193:47731 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:23.012661 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:23.006893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:56559)
0808 18:34:23.007262 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:23.007275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:23.007417 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:23.007821 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:23.007824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:23.008060 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:23.008163 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.008169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.009039 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.009041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.009585 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.009592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.010074 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.010076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.010139 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.010644 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:23.010656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:23.012498 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":87,"threads_started":1}
W20260808 18:34:23.012884 6297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:23.006946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:56559 (local address 127.27.251.254:33047)
0808 18:34:23.007108 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:23.007112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:23.007366 (+ 254us) server_negotiation.cc:407] Connection header received
0808 18:34:23.007594 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:23.007598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:23.007661 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:23.007752 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:23.008252 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.008940 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.009735 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.009966 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.010056 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.012754 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:56559: BlockingRecv error: recv got EOF from 127.27.251.194:56559 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:23.241637 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:23.235506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:37877)
0808 18:34:23.235816 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:23.235830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:23.235933 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:23.236198 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:23.236200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:23.236385 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:23.236486 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.236492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.237319 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.237321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.237893 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.237902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.238409 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.238412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.238476 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.239093 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:23.239106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:23.241386 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":125,"threads_started":1}
W20260808 18:34:23.241674 6297 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:23.235608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:37877 (local address 127.27.251.254:33047)
0808 18:34:23.235749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:23.235753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:23.235902 (+ 149us) server_negotiation.cc:407] Connection header received
0808 18:34:23.236015 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:23.236017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:23.236081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:23.236151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:23.236653 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.237208 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.238041 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.238296 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.238396 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.241557 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:37877: BlockingRecv error: recv got EOF from 127.27.251.195:37877 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:23 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:23.815660 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:23.810092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:55443)
0808 18:34:23.810448 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:23.810464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:23.810588 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:23.810921 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:23.810923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:23.811117 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:23.811219 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.811225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.811965 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.811967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.812621 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:23.812630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.813043 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.813045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:23.813109 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.813674 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:23.813687 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:23.815511 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":89,"threads_started":1}
W20260808 18:34:23.815804 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:23.810168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:55443 (local address 127.27.251.254:33047)
0808 18:34:23.810446 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:23.810450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:23.810531 (+ 81us) server_negotiation.cc:407] Connection header received
0808 18:34:23.810706 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:23.810712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:23.810774 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:23.810869 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:23.811326 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.811870 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.812720 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:23.812945 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:23.813023 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:23.815667 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:55443: BlockingRecv error: recv got EOF from 127.27.251.193:55443 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:24.019234 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:24.013515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:47399)
0808 18:34:24.013893 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:24.013906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:24.014064 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:24.014364 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:24.014367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:24.014582 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:24.014699 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.014705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.015483 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.015488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.016057 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.016065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.016587 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.016590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.016697 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.017202 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:24.017214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:24.019078 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
W20260808 18:34:24.019321 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:24.013635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:47399 (local address 127.27.251.254:33047)
0808 18:34:24.013839 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:24.013845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:24.014007 (+ 162us) server_negotiation.cc:407] Connection header received
0808 18:34:24.014182 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:24.014186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:24.014240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:24.014338 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:24.014827 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.015365 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.016187 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.016419 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.016566 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.019228 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:47399: BlockingRecv error: recv got EOF from 127.27.251.194:47399 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:24.248730 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:24.242667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:34279)
0808 18:34:24.242970 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:24.242987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:24.243092 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:24.243353 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:24.243355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:24.243535 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:24.243646 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.243652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.244620 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.244624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.245452 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.245462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.245908 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.245910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.245971 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.246548 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:24.246560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:24.248580 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
W20260808 18:34:24.248852 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:24.242864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:34279 (local address 127.27.251.254:33047)
0808 18:34:24.243043 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:24.243048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:24.243064 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:24.243185 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:24.243187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:24.243238 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:24.243309 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:24.243777 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.244489 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.245586 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.245804 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.245889 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.248722 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:34279: BlockingRecv error: recv got EOF from 127.27.251.195:34279 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:24 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:24.822499 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:24.816685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:33903)
0808 18:34:24.817106 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:24.817120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:24.817222 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:24.817488 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:24.817490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:24.817691 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:24.817794 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.817799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.818523 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.818525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.819100 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:24.819109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.819570 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.819574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:24.819663 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.820301 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:24.820317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:24.822379 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":86,"threads_started":1}
W20260808 18:34:24.822665 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:24.816784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:33903 (local address 127.27.251.254:33047)
0808 18:34:24.817161 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:24.817165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:24.817179 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:24.817310 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:24.817312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:24.817359 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:24.817441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:24.817898 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.818425 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.819202 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:24.819438 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:24.819555 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:24.822516 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:33903: BlockingRecv error: recv got EOF from 127.27.251.193:33903 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":81,"threads_started":1}
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:25.026528 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:25.020280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:45297)
0808 18:34:25.020733 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:25.020746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:25.020893 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:25.021213 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:25.021215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:25.021442 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:25.021540 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.021546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.022373 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.022378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.023026 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.023035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.023536 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.023540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.023630 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.024236 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:25.024249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:25.026350 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":90,"threads_started":1}
W20260808 18:34:25.026664 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:25.020345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:45297 (local address 127.27.251.254:33047)
0808 18:34:25.020541 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:25.020548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:25.020828 (+ 280us) server_negotiation.cc:407] Connection header received
0808 18:34:25.021033 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:25.021036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:25.021090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:25.021194 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:25.021670 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.022252 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.023155 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.023404 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.023532 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.026542 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:45297: BlockingRecv error: recv got EOF from 127.27.251.194:45297 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:25.255774 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:25.249709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:55097)
0808 18:34:25.250005 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:25.250018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:25.250121 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:25.250438 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:25.250444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:25.250661 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:25.250769 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.250775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.251569 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.251572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.252181 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.252190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.252772 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.252775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.252844 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.253470 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:25.253487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:25.255590 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260808 18:34:25.255913 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:25.249839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:55097 (local address 127.27.251.254:33047)
0808 18:34:25.249999 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:25.250002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:25.250071 (+ 69us) server_negotiation.cc:407] Connection header received
0808 18:34:25.250223 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:25.250225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:25.250275 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:25.250340 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:25.250901 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.251448 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.252300 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.252639 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.252753 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.255767 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:55097: BlockingRecv error: recv got EOF from 127.27.251.195:55097 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:25 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:25.829016 6308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:25.823352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:56767)
0808 18:34:25.823654 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:25.823676 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:25.823772 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:25.824053 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:25.824056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:25.824271 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:25.824374 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.824380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.825261 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.825267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.825824 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:25.825832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.826355 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.826360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:25.826486 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.826989 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:25.827001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:25.828899 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
W20260808 18:34:25.829218 6309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:25.823484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56767 (local address 127.27.251.254:33047)
0808 18:34:25.823757 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:25.823760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:25.823774 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:25.823843 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:25.823845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:25.823888 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:25.823981 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:25.824588 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.825141 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.825948 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:25.826195 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:25.826326 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:25.829041 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:56767: BlockingRecv error: recv got EOF from 127.27.251.193:56767 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:26.033300 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:26.027415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:42095)
0808 18:34:26.027734 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:26.027748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:26.027883 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:26.028112 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:26.028114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:26.028307 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:26.028407 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.028412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.029153 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.029156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.029721 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.029729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.030263 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.030267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.030347 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.030941 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:26.030954 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:26.033161 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
W20260808 18:34:26.033536 6309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:26.027577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42095 (local address 127.27.251.254:33047)
0808 18:34:26.027740 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:26.027744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:26.027866 (+ 122us) server_negotiation.cc:407] Connection header received
0808 18:34:26.027975 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:26.027977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:26.028028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:26.028097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:26.028529 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.029044 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.029837 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.030062 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.030157 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.033369 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:42095: BlockingRecv error: recv got EOF from 127.27.251.194:42095 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:26.267125 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:26.256855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:50927)
0808 18:34:26.257174 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:26.257191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:26.257359 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:26.257671 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:26.257674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:26.257939 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:26.258091 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.258102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.258862 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.258867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.259439 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.259447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.259936 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.259940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.260027 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.265006 (+ 4979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:26.265026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:26.266945 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
W20260808 18:34:26.267223 6309 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:26.257007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:50927 (local address 127.27.251.254:33047)
0808 18:34:26.257175 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:26.257182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:26.257291 (+ 109us) server_negotiation.cc:407] Connection header received
0808 18:34:26.257477 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:26.257485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:26.257533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:26.257619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:26.258209 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.258748 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.259565 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.259803 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.259917 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.267089 (+ 7172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:50927: BlockingRecv error: recv got EOF from 127.27.251.195:50927 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:26 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:26.835456 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:26.829884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:43949)
0808 18:34:26.830199 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:26.830213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:26.830308 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:26.830562 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:26.830564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:26.830765 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:26.830868 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.830873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.831617 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.831619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.832179 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:26.832185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.832672 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.832675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:26.832743 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.833319 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:26.833332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:26.835232 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
W20260808 18:34:26.835487 6313 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:26.830015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:43949 (local address 127.27.251.254:33047)
0808 18:34:26.830267 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:26.830271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:26.830284 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:34:26.830400 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:26.830402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:26.830448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:26.830526 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:26.830970 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.831501 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.832286 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:26.832549 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:26.832629 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:26.835358 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:43949: BlockingRecv error: recv got EOF from 127.27.251.193:43949 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:27.040740 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:27.034187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:33117)
0808 18:34:27.034558 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:27.034576 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:27.034725 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:27.035032 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:27.035035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:27.035262 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:27.035361 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.035367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.036133 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.036139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.036884 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.036894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.037420 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.037423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.037563 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.038326 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:27.038339 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:27.040618 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
W20260808 18:34:27.040928 6313 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:27.034390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:33117 (local address 127.27.251.254:33047)
0808 18:34:27.034623 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:27.034629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:27.034644 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:27.034839 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:27.034843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:27.034897 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:27.035003 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:27.035489 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.036018 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.037005 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.037243 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.037374 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.040763 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:33117: BlockingRecv error: recv got EOF from 127.27.251.194:33117 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:27.274150 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:27.267935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:35745)
0808 18:34:27.268309 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:27.268326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:27.268495 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:27.268819 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:27.268822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:27.269043 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:27.269140 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.269146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.269983 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.269989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.270601 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.270609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.271094 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.271097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.271185 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.271833 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:27.271846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:27.273981 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
W20260808 18:34:27.274431 6313 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:27.268089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:35745 (local address 127.27.251.254:33047)
0808 18:34:27.268272 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:27.268277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:27.268419 (+ 142us) server_negotiation.cc:407] Connection header received
0808 18:34:27.268629 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:27.268632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:27.268693 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:27.268787 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:27.269269 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.269832 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.270732 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.270973 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.271086 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.274181 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:35745: BlockingRecv error: recv got EOF from 127.27.251.195:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:27 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:27.842545 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:27.836407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:35575)
0808 18:34:27.836794 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:27.836810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:27.836915 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:27.837256 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:27.837259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:27.837468 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:27.837594 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.837600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.838368 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.838371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.839000 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:27.839009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.839515 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.839518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:27.839602 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.840135 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:27.840150 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:27.842411 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":112,"threads_started":1}
W20260808 18:34:27.842741 6317 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:27.836448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:35575 (local address 127.27.251.254:33047)
0808 18:34:27.836836 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:27.836843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:27.836874 (+ 31us) server_negotiation.cc:407] Connection header received
0808 18:34:27.837028 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:27.837031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:27.837104 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:27.837205 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:27.837704 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.838263 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.839110 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:27.839347 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:27.839423 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:27.842559 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:35575: BlockingRecv error: recv got EOF from 127.27.251.193:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":71,"threads_started":1}
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:28.048818 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:28.041745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:59525)
0808 18:34:28.042061 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:28.042079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:28.042217 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:28.042714 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:28.042716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:28.042905 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:28.043024 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.043030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.044347 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.044350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.044984 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.044994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.045733 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.045736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.045807 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.046548 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:28.046570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:28.048681 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":108,"threads_started":1}
W20260808 18:34:28.049088 6317 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:28.041893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:59525 (local address 127.27.251.254:33047)
0808 18:34:28.042102 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:28.042109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:28.042164 (+ 55us) server_negotiation.cc:407] Connection header received
0808 18:34:28.042328 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:28.042334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:28.042407 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:28.042513 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:28.043162 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.044199 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.045118 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.045581 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.045718 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.048908 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:59525: BlockingRecv error: recv got EOF from 127.27.251.194:59525 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.195@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:28.281736 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:28.275060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.195:60553)
0808 18:34:28.275381 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:28.275396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:28.275512 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:28.275863 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:28.275867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:28.276138 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:28.276311 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.276326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.277530 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.277534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.278185 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.278195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.278752 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.278755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.278831 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.279402 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:28.279424 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:28.281531 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
W20260808 18:34:28.281774 6317 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:28.275169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:60553 (local address 127.27.251.254:33047)
0808 18:34:28.275321 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:28.275325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:28.275477 (+ 152us) server_negotiation.cc:407] Connection header received
0808 18:34:28.275660 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:28.275665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:28.275721 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:28.275834 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:28.276503 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.277348 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.278318 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.278604 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.278695 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.281660 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.195:60553: BlockingRecv error: recv got EOF from 127.27.251.195:60553 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:28 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.193@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:28.850363 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:28.843932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.193:59885)
0808 18:34:28.844287 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:28.844307 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:28.844424 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:28.844808 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:28.844811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:28.845148 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:28.845282 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.845289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.846403 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.846406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.846973 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:28.846981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.847484 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.847487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:28.847557 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.848149 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:28.848161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:28.850238 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
W20260808 18:34:28.850574 6321 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:28.844049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:59885 (local address 127.27.251.254:33047)
0808 18:34:28.844367 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:28.844371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:28.844388 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:34:28.844570 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:28.844574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:28.844636 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:28.844784 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:28.845538 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.846270 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.847102 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:28.847351 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:28.847456 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:28.850393 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.193:59885: BlockingRecv error: recv got EOF from 127.27.251.193:59885 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
Aug 08 18:34:29 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 18:34:29 dist-test-slave-fd7l krb5kdc[5178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.251.194@KRBTEST.COM for oryx/127.27.251.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 18:34:29.055739 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.050137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:33047 (local address 127.27.251.194:59347)
0808 18:34:29.050468 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:29.050483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:29.050594 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:29.050853 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:29.050856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:29.051049 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:29.051164 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.051169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.051888 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.051890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.052453 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.052459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.052909 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.052911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.052978 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.053550 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:29.053562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:29.055552 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.251.254:33047: Server oryx/127.27.251.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
W20260808 18:34:29.055845 6321 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:29.050252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:59347 (local address 127.27.251.254:33047)
0808 18:34:29.050472 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:29.050476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:29.050564 (+ 88us) server_negotiation.cc:407] Connection header received
0808 18:34:29.050686 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:29.050689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:29.050735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:29.050808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:29.051271 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.051792 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.052584 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.052805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.052881 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.055721 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.251.194:59347: BlockingRecv error: recv got EOF from 127.27.251.194:59347 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 18:34:29.159659 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5814
I20260808 18:34:29.169385 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5979
I20260808 18:34:29.176723 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6119
I20260808 18:34:29.184155 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 5736
2026-08-08T18:34:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18129 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T18:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:29Z Disabled control of system clock
I20260808 18:34:29.218922 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35915
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 18:34:29.335681 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:29.335961 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:29.335996 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:29.339985 6326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:29.340046 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:29.340062 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:29.340080 6326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:29.340095 6326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:29.344980 6326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:29.346122 6326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:29.347510 6326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:29.357072 6331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:29.357125 6326 server_base.cc:1034] running on GCE node
W20260808 18:34:29.357138 6334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:29.357147 6332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:29.357923 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:29.359120 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:29.360333 6326 hybrid_clock.cc:648] HybridClock initialized: now 1786214069360306 us; error 27 us; skew 500 ppm
I20260808 18:34:29.362537 6326 webserver.cc:460] Webserver started at http://127.27.251.254:40639/ using document root <none> and password file <none>
I20260808 18:34:29.363152 6326 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:29.363250 6326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:29.363489 6326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:29.365669 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "0cc9950a6fdb41ccac25bda91c7a61fc"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "84c039b55eba1ab2fe5927b1be0fdc98"
server_key_iv: "19208bc9ec976761d4e5c8555c99657a"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.366259 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "0cc9950a6fdb41ccac25bda91c7a61fc"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "84c039b55eba1ab2fe5927b1be0fdc98"
server_key_iv: "19208bc9ec976761d4e5c8555c99657a"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.370066 6326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:29.372464 6341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.373714 6326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:29.373857 6326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "0cc9950a6fdb41ccac25bda91c7a61fc"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "84c039b55eba1ab2fe5927b1be0fdc98"
server_key_iv: "19208bc9ec976761d4e5c8555c99657a"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.373970 6326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:29.387171 6326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:29.387857 6326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:29.388068 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:29.395466 6326 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35915
I20260808 18:34:29.395485 6397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35915 every 8 connection(s)
I20260808 18:34:29.396481 6326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:29.399943 6398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:29.405735 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc: Bootstrap starting.
I20260808 18:34:29.406142 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6326
I20260808 18:34:29.406268 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:29.406587 28655 external_mini_cluster.cc:1428] Setting key aeea139f74903098d4730d9b9425f6b2
I20260808 18:34:29.408633 6398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:29.409389 6398 log.cc:824] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:29.411293 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc: No bootstrap required, opened a new log
I20260808 18:34:29.414898 6398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } }
I20260808 18:34:29.415138 6398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:29.415216 6398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc9950a6fdb41ccac25bda91c7a61fc, State: Initialized, Role: FOLLOWER
I20260808 18:34:29.415705 6398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } }
I20260808 18:34:29.415851 6398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:29.415935 6398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:29.416038 6398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:29.416371 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.408030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.27.251.254:35915)
0808 18:34:29.408876 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:29.408893 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:29.408930 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:34:29.409631 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:29.409650 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:29.410051 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:29.410382 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:29.410821 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.411871 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.412777 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.413253 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.413429 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.414391 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:29.414397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:29.414409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:29.414793 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:29.414847 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:29.414867 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:29.414974 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:29.415158 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 18:34:29.415439 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":151,"threads_started":1}
I20260808 18:34:29.417014 6398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } }
I20260808 18:34:29.417351 6398 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc9950a6fdb41ccac25bda91c7a61fc; no voters:
I20260808 18:34:29.417668 6398 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:29.417759 6403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:29.417972 6403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [term 1 LEADER]: Becoming Leader. State: Replica: 0cc9950a6fdb41ccac25bda91c7a61fc, State: Running, Role: LEADER
I20260808 18:34:29.418346 6403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } }
I20260808 18:34:29.418869 6398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:29.419668 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc9950a6fdb41ccac25bda91c7a61fc. Latest consensus state: current_term: 1 leader_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } } }
I20260808 18:34:29.419809 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:29.420083 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc9950a6fdb41ccac25bda91c7a61fc" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35915 } } }
I20260808 18:34:29.420178 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:29.420548 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:29.423568 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:29.429715 6412 catalog_manager.cc:1347] Generated new cluster ID: f1457a19e1734e2e8cb11a9a56a174a5
I20260808 18:34:29.429800 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:29.446985 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:29.447928 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:29.462670 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc9950a6fdb41ccac25bda91c7a61fc: Generated new TSK 0
I20260808 18:34:29.463748 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:34:29.468173 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--builtin_ntp_servers=127.27.251.212:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 18:34:29.586146 6422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:29.586421 6422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:29.586458 6422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:29.590528 6422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:29.590585 6422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:29.590639 6422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:29.595372 6422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:29.596592 6422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:29.598047 6422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:29.607723 6429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:29.607692 6432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:29.607780 6428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:29.607898 6422 server_base.cc:1034] running on GCE node
I20260808 18:34:29.608353 6422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:29.609032 6422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:29.610217 6422 hybrid_clock.cc:648] HybridClock initialized: now 1786214069610182 us; error 43 us; skew 500 ppm
I20260808 18:34:29.612118 6422 webserver.cc:460] Webserver started at http://127.27.251.193:44815/ using document root <none> and password file <none>
I20260808 18:34:29.612783 6422 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:29.612879 6422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:29.613113 6422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:29.615134 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "c18878f7c5304cd8bc3b5a6b51ac4d85"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "636bff49204d272c0dae80e93b9631eb"
server_key_iv: "412b71e1d9d53684514b2f5d53c86e80"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.615706 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "c18878f7c5304cd8bc3b5a6b51ac4d85"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "636bff49204d272c0dae80e93b9631eb"
server_key_iv: "412b71e1d9d53684514b2f5d53c86e80"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.619398 6422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 18:34:29.622085 6438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.623337 6422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:29.623484 6422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c18878f7c5304cd8bc3b5a6b51ac4d85"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "636bff49204d272c0dae80e93b9631eb"
server_key_iv: "412b71e1d9d53684514b2f5d53c86e80"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.623610 6422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:29.648942 6422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:29.649682 6422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:29.649891 6422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:29.650499 6422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:29.651558 6422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:29.651638 6422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.651710 6422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:29.651762 6422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.660400 6422 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:42147
I20260808 18:34:29.660513 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:42147 every 8 connection(s)
I20260808 18:34:29.661366 6422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:29.666623 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6422
I20260808 18:34:29.666772 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:29.667081 28655 external_mini_cluster.cc:1428] Setting key 4941d5630a670d062784aac311bc1bc1
I20260808 18:34:29.669222 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--builtin_ntp_servers=127.27.251.212:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 18:34:29.671437 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.663387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:34625 (local address 127.27.251.254:35915)
0808 18:34:29.663580 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:29.663585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:29.664673 (+ 1088us) server_negotiation.cc:407] Connection header received
0808 18:34:29.666013 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:29.666016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:29.666062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:29.666141 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:29.667196 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.667749 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.669093 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.669396 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.669505 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.670825 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:29.670828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:29.670831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:29.670842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:29.670861 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:29.670869 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:29.670929 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:29.671157 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 18:34:29.671239 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 18:34:29.672331 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.663811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35915 (local address 127.27.251.193:34625)
0808 18:34:29.664453 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:29.664492 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:29.665829 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:29.666298 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:29.666305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:29.666403 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:29.667043 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.667059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.667883 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.667887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.668953 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.668963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.669572 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.669579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.669830 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.670498 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:29.670503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:29.670561 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:29.670658 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:29.670957 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:29.670970 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:29.670979 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:34:29.671127 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 18:34:29.671418 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260808 18:34:29.673681 6559 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35915
I20260808 18:34:29.673920 6559 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:29.674530 6559 heartbeater.cc:511] Master 127.27.251.254:35915 requested a full tablet report, sending...
I20260808 18:34:29.676065 6359 ts_manager.cc:194] Registered new tserver with Master: c18878f7c5304cd8bc3b5a6b51ac4d85 (127.27.251.193:42147)
I20260808 18:34:29.677418 6359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.193:34625
W20260808 18:34:29.786650 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:29.786909 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:29.786947 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:29.791127 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:29.791198 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:29.791252 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:29.796032 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:29.797264 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:29.798692 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:29.808660 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:29.808653 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:29.808743 6562 server_base.cc:1034] running on GCE node
W20260808 18:34:29.808661 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:29.809386 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:29.810041 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:29.811208 6562 hybrid_clock.cc:648] HybridClock initialized: now 1786214069811187 us; error 31 us; skew 500 ppm
I20260808 18:34:29.813247 6562 webserver.cc:460] Webserver started at http://127.27.251.194:41027/ using document root <none> and password file <none>
I20260808 18:34:29.813871 6562 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:29.813936 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:29.814180 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:29.816291 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "bb1432b0acb84114b95bdba33e07a51d"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "701afc4b42f27f90417b8199a26ed6cf"
server_key_iv: "33228d8dc4925c31286b0ee0b65540fe"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.816977 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "bb1432b0acb84114b95bdba33e07a51d"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "701afc4b42f27f90417b8199a26ed6cf"
server_key_iv: "33228d8dc4925c31286b0ee0b65540fe"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.821015 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:34:29.823551 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.824906 6562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:29.825037 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "bb1432b0acb84114b95bdba33e07a51d"
format_stamp: "Formatted at 2026-08-08 18:34:29 on dist-test-slave-fd7l"
server_key: "701afc4b42f27f90417b8199a26ed6cf"
server_key_iv: "33228d8dc4925c31286b0ee0b65540fe"
server_key_version: "encryptionkey@0"
I20260808 18:34:29.825150 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:29.841306 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:29.842072 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:29.842298 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:29.842934 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:29.843919 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:29.843966 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.844038 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:29.844071 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:29.853193 6562 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:46655
I20260808 18:34:29.853238 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:46655 every 8 connection(s)
I20260808 18:34:29.854122 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:29.857745 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6562
I20260808 18:34:29.857878 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:29.858204 28655 external_mini_cluster.cc:1428] Setting key 5a30d66168d855ba6b51abb38844fce5
I20260808 18:34:29.860637 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--builtin_ntp_servers=127.27.251.212:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 18:34:29.864579 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.856056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42765 (local address 127.27.251.254:35915)
0808 18:34:29.856205 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:29.856209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:29.857328 (+ 1119us) server_negotiation.cc:407] Connection header received
0808 18:34:29.858839 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:29.858843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:29.858901 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:29.858995 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:29.860212 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.861045 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.862213 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.862536 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.862619 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.864065 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:29.864068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:29.864070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:29.864082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:29.864100 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:29.864108 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:29.864167 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:29.864358 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 18:34:29.864408 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:34:29.865808 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:29.856434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35915 (local address 127.27.251.194:42765)
0808 18:34:29.857173 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:29.857215 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:29.858621 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:29.859229 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:29.859238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:29.859344 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:29.860050 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.860073 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.861172 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.861174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.862068 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:29.862077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:29.862706 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:29.862712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:29.862959 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:29.863737 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:29.863740 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:29.863799 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:29.863890 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:29.864196 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:29.864209 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:29.864218 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:34:29.864351 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 18:34:29.864682 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":157,"threads_started":1}
I20260808 18:34:29.867087 6698 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35915
I20260808 18:34:29.867337 6698 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:29.867911 6698 heartbeater.cc:511] Master 127.27.251.254:35915 requested a full tablet report, sending...
I20260808 18:34:29.869019 6359 ts_manager.cc:194] Registered new tserver with Master: bb1432b0acb84114b95bdba33e07a51d (127.27.251.194:46655)
I20260808 18:34:29.869622 6359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.194:42765
W20260808 18:34:29.978523 6701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:29.978811 6701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:29.978880 6701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:29.982899 6701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:29.982978 6701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:29.983074 6701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:29.987808 6701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:29.988993 6701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:29.990406 6701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:30.000128 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.000174 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.000134 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.000342 6701 server_base.cc:1034] running on GCE node
I20260808 18:34:30.000866 6701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:30.001580 6701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:30.002774 6701 hybrid_clock.cc:648] HybridClock initialized: now 1786214070002750 us; error 33 us; skew 500 ppm
I20260808 18:34:30.004832 6701 webserver.cc:460] Webserver started at http://127.27.251.195:38191/ using document root <none> and password file <none>
I20260808 18:34:30.005416 6701 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:30.005482 6701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:30.005666 6701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:30.007915 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "cf1d2eed06624b3cba6ed1fbc58fadb5"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "c0c14a05f4ecad0eab10289885185bf2"
server_key_iv: "e47b74e0bfb67a1303bbdda5b20f83c0"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.008652 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "cf1d2eed06624b3cba6ed1fbc58fadb5"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "c0c14a05f4ecad0eab10289885185bf2"
server_key_iv: "e47b74e0bfb67a1303bbdda5b20f83c0"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.012480 6701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:34:30.015185 6715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.016453 6701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:34:30.016636 6701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "cf1d2eed06624b3cba6ed1fbc58fadb5"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "c0c14a05f4ecad0eab10289885185bf2"
server_key_iv: "e47b74e0bfb67a1303bbdda5b20f83c0"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.016739 6701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:30.029613 6701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:30.030270 6701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:30.030417 6701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:30.030928 6701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:30.031802 6701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:30.031847 6701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.031886 6701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:30.031908 6701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.041191 6701 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:34681
I20260808 18:34:30.041224 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:34681 every 8 connection(s)
I20260808 18:34:30.042074 6701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:30.048810 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6701
I20260808 18:34:30.048931 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:30.049258 28655 external_mini_cluster.cc:1428] Setting key eaeb602fdec68724813a02b2af3271d8
I20260808 18:34:30.052413 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.044138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:40777 (local address 127.27.251.254:35915)
0808 18:34:30.044313 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.044318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.045388 (+ 1070us) server_negotiation.cc:407] Connection header received
0808 18:34:30.046807 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.046810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.046861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.046937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.047807 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:30.048345 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:30.049942 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:30.050282 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:30.050404 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:30.051892 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:30.051896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:30.051898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:30.051913 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:30.051940 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:30.051948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:30.051985 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:30.052203 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 18:34:30.052269 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 18:34:30.053290 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.044573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35915 (local address 127.27.251.195:40777)
0808 18:34:30.045227 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:30.045270 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:30.046603 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:30.047077 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:30.047084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:30.047168 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:30.047674 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:30.047688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:30.048461 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:30.048463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:30.049830 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:30.049841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:30.050446 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:30.050450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:30.050668 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:30.051532 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:30.051534 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:30.051595 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:30.051712 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:30.052052 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:30.052060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:30.052064 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:34:30.052191 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 18:34:30.052484 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260808 18:34:30.054373 6837 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35915
I20260808 18:34:30.054625 6837 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:30.055166 6837 heartbeater.cc:511] Master 127.27.251.254:35915 requested a full tablet report, sending...
I20260808 18:34:30.056166 6359 ts_manager.cc:194] Registered new tserver with Master: cf1d2eed06624b3cba6ed1fbc58fadb5 (127.27.251.195:34681)
I20260808 18:34:30.056803 6359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.195:40777
I20260808 18:34:30.063606 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:34:30.066766 6401 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:30.065737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.27.251.254:35915)
0808 18:34:30.065887 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.065893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.066080 (+ 187us) server_negotiation.cc:407] Connection header received
0808 18:34:30.066238 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.066241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.066293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.066374 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.066631 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43318: BlockingRecv error: recv got EOF from 127.0.0.1:43318 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:34:30.067212 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6422
I20260808 18:34:30.076543 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6562
I20260808 18:34:30.084930 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6701
I20260808 18:34:30.096447 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6326
2026-08-08T18:34:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (911 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T18:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:30Z Disabled control of system clock
I20260808 18:34:30.129817 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34105
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:34:30.245916 6849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:30.246198 6849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:30.246237 6849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:30.250203 6849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:30.250264 6849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:30.250280 6849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:30.250296 6849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:30.250312 6849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:30.255144 6849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:34105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:30.256305 6849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:30.257833 6849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:30.267129 6854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.267232 6849 server_base.cc:1034] running on GCE node
W20260808 18:34:30.267201 6857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.267139 6855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.267984 6849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:30.269251 6849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:30.270421 6849 hybrid_clock.cc:648] HybridClock initialized: now 1786214070270387 us; error 52 us; skew 500 ppm
I20260808 18:34:30.272498 6849 webserver.cc:460] Webserver started at http://127.27.251.254:44615/ using document root <none> and password file <none>
I20260808 18:34:30.273182 6849 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:30.273249 6849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:30.273530 6849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:30.275681 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "09bbab7615ee42d5935f10222fa5956f"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "49f06b2b0385653370632f2140b78fc9"
server_key_iv: "01531804fe40ea31b1935d9bbb72313e"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.276270 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "09bbab7615ee42d5935f10222fa5956f"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "49f06b2b0385653370632f2140b78fc9"
server_key_iv: "01531804fe40ea31b1935d9bbb72313e"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.280105 6849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 18:34:30.282660 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.283931 6849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:34:30.284078 6849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "09bbab7615ee42d5935f10222fa5956f"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "49f06b2b0385653370632f2140b78fc9"
server_key_iv: "01531804fe40ea31b1935d9bbb72313e"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.284196 6849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:30.288818 6849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:30.289438 6849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:30.289634 6849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:30.297477 6849 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:34105
I20260808 18:34:30.297513 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:34105 every 8 connection(s)
I20260808 18:34:30.298516 6849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:30.302098 6920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:30.305841 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6849
I20260808 18:34:30.305966 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:30.306296 28655 external_mini_cluster.cc:1428] Setting key 63da410129af4f195a49050b6a9da5e3
I20260808 18:34:30.308688 6920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f: Bootstrap starting.
I20260808 18:34:30.311057 6920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:30.311924 6920 log.cc:824] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:30.312196 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.307857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.27.251.254:34105)
0808 18:34:30.308622 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.308639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.308676 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:34:30.309376 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.309394 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.309753 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.310094 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.310903 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:30.310907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:30.310919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:30.310930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:30.310985 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:30.311001 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:30.311118 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:30.311237 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 18:34:30.311474 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":157,"threads_started":1}
I20260808 18:34:30.314108 6920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f: No bootstrap required, opened a new log
I20260808 18:34:30.317588 6920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } }
I20260808 18:34:30.317848 6920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:30.317914 6920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bbab7615ee42d5935f10222fa5956f, State: Initialized, Role: FOLLOWER
I20260808 18:34:30.318354 6920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } }
I20260808 18:34:30.318482 6920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:30.318531 6920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:30.318602 6920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:30.319420 6920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } }
I20260808 18:34:30.319705 6920 leader_election.cc:302] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bbab7615ee42d5935f10222fa5956f; no voters:
I20260808 18:34:30.319938 6920 leader_election.cc:288] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:30.320055 6925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:30.320298 6925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [term 1 LEADER]: Becoming Leader. State: Replica: 09bbab7615ee42d5935f10222fa5956f, State: Running, Role: LEADER
I20260808 18:34:30.320686 6925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } }
I20260808 18:34:30.321182 6920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:30.322271 6926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bbab7615ee42d5935f10222fa5956f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } } }
I20260808 18:34:30.322400 6926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:30.322477 6927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bbab7615ee42d5935f10222fa5956f. Latest consensus state: current_term: 1 leader_uuid: "09bbab7615ee42d5935f10222fa5956f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bbab7615ee42d5935f10222fa5956f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34105 } } }
I20260808 18:34:30.322585 6927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:30.322831 6935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:30.325987 6935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:30.331593 6935 catalog_manager.cc:1347] Generated new cluster ID: 831011e856144a33969b4f3892b3ea3e
I20260808 18:34:30.331676 6935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:30.355180 6935 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:30.355553 6935 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:30.361629 6935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bbab7615ee42d5935f10222fa5956f: Generated new TSK 0
I20260808 18:34:30.362334 6935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:34:30.364897 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--builtin_ntp_servers=127.27.251.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:34:30.498412 6945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:30.498703 6945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:30.498746 6945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:30.502948 6945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:30.503005 6945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:30.503063 6945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:30.507750 6945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:30.508934 6945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:30.510329 6945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:30.520272 6952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.520241 6954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.520280 6951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.520587 6945 server_base.cc:1034] running on GCE node
I20260808 18:34:30.521065 6945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:30.521775 6945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:30.522943 6945 hybrid_clock.cc:648] HybridClock initialized: now 1786214070522911 us; error 41 us; skew 500 ppm
I20260808 18:34:30.525039 6945 webserver.cc:460] Webserver started at http://127.27.251.193:41889/ using document root <none> and password file <none>
I20260808 18:34:30.525679 6945 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:30.525746 6945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:30.526021 6945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:30.528091 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "4c8d993b27a94f2f89589656a4946346"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "0a0abf4db1e56df7bad56b2a15fc7343"
server_key_iv: "7c736e7721e92ce71d50ca43fb599a46"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.528719 6945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "4c8d993b27a94f2f89589656a4946346"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "0a0abf4db1e56df7bad56b2a15fc7343"
server_key_iv: "7c736e7721e92ce71d50ca43fb599a46"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.532573 6945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:34:30.535323 6960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.536829 6945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:34:30.536991 6945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "4c8d993b27a94f2f89589656a4946346"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "0a0abf4db1e56df7bad56b2a15fc7343"
server_key_iv: "7c736e7721e92ce71d50ca43fb599a46"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.537132 6945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:30.541980 6945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:30.542634 6945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:30.542833 6945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:30.543404 6945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:30.544341 6945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:30.544413 6945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.544484 6945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:30.544564 6945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.553139 6945 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37165
I20260808 18:34:30.553191 7081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37165 every 8 connection(s)
I20260808 18:34:30.554147 6945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:30.560407 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.555982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:52269 (local address 127.27.251.254:34105)
0808 18:34:30.556137 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.556141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.557253 (+ 1112us) server_negotiation.cc:407] Connection header received
0808 18:34:30.558601 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.558604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.558652 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.558744 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.559917 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:30.559920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:30.559923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:30.559935 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:30.559956 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:30.559972 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:30.560072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:30.560223 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 18:34:30.560271 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:34:30.561290 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.556379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34105 (local address 127.27.251.193:52269)
0808 18:34:30.557093 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:30.557135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:30.558379 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:30.558926 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:30.558933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:30.559020 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:30.559645 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:30.559646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:30.559689 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:30.559757 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:30.560055 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:30.560065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:30.560070 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:34:30.560266 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 18:34:30.560510 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":166,"threads_started":1}
I20260808 18:34:30.562526 7083 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34105
I20260808 18:34:30.562768 7083 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:30.563241 7083 heartbeater.cc:511] Master 127.27.251.254:34105 requested a full tablet report, sending...
I20260808 18:34:30.563338 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 6945
I20260808 18:34:30.563453 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:30.563823 28655 external_mini_cluster.cc:1428] Setting key 202095679bcf47dd90ff41003fd65969
I20260808 18:34:30.564968 6882 ts_manager.cc:194] Registered new tserver with Master: 4c8d993b27a94f2f89589656a4946346 (127.27.251.193:37165)
I20260808 18:34:30.566002 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--builtin_ntp_servers=127.27.251.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:34:30.679392 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:30.679695 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:30.679770 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:30.683815 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:30.683907 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:30.684002 7087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:30.689100 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:30.690253 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:30.691973 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:30.702867 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.702867 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.702942 7087 server_base.cc:1034] running on GCE node
W20260808 18:34:30.702895 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.703594 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:30.704265 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:30.705430 7087 hybrid_clock.cc:648] HybridClock initialized: now 1786214070705392 us; error 49 us; skew 500 ppm
I20260808 18:34:30.707281 7087 webserver.cc:460] Webserver started at http://127.27.251.194:39103/ using document root <none> and password file <none>
I20260808 18:34:30.707885 7087 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:30.707949 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:30.708223 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:30.710376 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "fccbfa837d654acfbac2742f2bc5396b"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "79aadabc5ffbfcbeb58a5e6ebfd3245a"
server_key_iv: "08e58d1a5fc58f027b735b21aa1cc4fc"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.711025 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "fccbfa837d654acfbac2742f2bc5396b"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "79aadabc5ffbfcbeb58a5e6ebfd3245a"
server_key_iv: "08e58d1a5fc58f027b735b21aa1cc4fc"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.714814 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:30.717406 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.718616 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 18:34:30.718746 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "fccbfa837d654acfbac2742f2bc5396b"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "79aadabc5ffbfcbeb58a5e6ebfd3245a"
server_key_iv: "08e58d1a5fc58f027b735b21aa1cc4fc"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.718852 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:30.723812 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:30.724421 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:30.724615 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:30.725168 7087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:30.726055 7087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:30.726102 7087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.726148 7087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:30.726171 7087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.734512 7087 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:45043
I20260808 18:34:30.734561 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:45043 every 8 connection(s)
I20260808 18:34:30.735405 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:30.741470 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.737291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:39075 (local address 127.27.251.254:34105)
0808 18:34:30.737435 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.737439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.738538 (+ 1099us) server_negotiation.cc:407] Connection header received
0808 18:34:30.739797 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.739799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.739837 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.739915 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.741030 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:30.741033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:30.741035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:30.741047 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:30.741069 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:30.741077 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:30.741169 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:30.741291 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 18:34:30.741352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:34:30.742324 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.737707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34105 (local address 127.27.251.194:39075)
0808 18:34:30.738377 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:30.738423 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:30.739614 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:30.740084 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:30.740090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:30.740177 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:30.740775 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:30.740777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:30.740817 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:30.740883 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:30.741156 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:30.741164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:30.741168 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:34:30.741298 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 18:34:30.741485 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260808 18:34:30.743285 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7087
I20260808 18:34:30.743485 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:30.743503 7220 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34105
I20260808 18:34:30.743757 7220 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:30.743858 28655 external_mini_cluster.cc:1428] Setting key 5380f09675d1d6949fa0744495f90e70
I20260808 18:34:30.744230 7220 heartbeater.cc:511] Master 127.27.251.254:34105 requested a full tablet report, sending...
I20260808 18:34:30.745404 6882 ts_manager.cc:194] Registered new tserver with Master: fccbfa837d654acfbac2742f2bc5396b (127.27.251.194:45043)
I20260808 18:34:30.746064 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--builtin_ntp_servers=127.27.251.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 18:34:30.876916 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:30.877343 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:30.877431 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:30.884764 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:30.884825 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:30.884917 7223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:30.889966 7223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:30.891155 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:30.892712 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:34:30.902943 7223 server_base.cc:1034] running on GCE node
W20260808 18:34:30.902974 7228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.902988 7231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:30.903090 7229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:30.903570 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:30.904215 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:30.905381 7223 hybrid_clock.cc:648] HybridClock initialized: now 1786214070905344 us; error 45 us; skew 500 ppm
I20260808 18:34:30.907197 7223 webserver.cc:460] Webserver started at http://127.27.251.195:41175/ using document root <none> and password file <none>
I20260808 18:34:30.907752 7223 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:30.907809 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:30.907974 7223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:30.910065 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "00f4b7f207454009b041dd1f6850cccb"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "e43c9f50dd1da65f51e23fb65a043c50"
server_key_iv: "c836e83112398ed2c60d17179cc36f68"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.910598 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "00f4b7f207454009b041dd1f6850cccb"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "e43c9f50dd1da65f51e23fb65a043c50"
server_key_iv: "c836e83112398ed2c60d17179cc36f68"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.914287 7223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260808 18:34:30.916882 7237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.918123 7223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:34:30.918252 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "00f4b7f207454009b041dd1f6850cccb"
format_stamp: "Formatted at 2026-08-08 18:34:30 on dist-test-slave-fd7l"
server_key: "e43c9f50dd1da65f51e23fb65a043c50"
server_key_iv: "c836e83112398ed2c60d17179cc36f68"
server_key_version: "encryptionkey@0"
I20260808 18:34:30.918366 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:30.923447 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:30.924093 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:30.924273 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:30.924875 7223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:30.925771 7223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:30.925818 7223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.925884 7223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:30.925921 7223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:30.934876 7223 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:37739
I20260808 18:34:30.934898 7356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:37739 every 8 connection(s)
I20260808 18:34:30.935819 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:30.942348 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.937737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:39789 (local address 127.27.251.254:34105)
0808 18:34:30.937875 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.937879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.938996 (+ 1117us) server_negotiation.cc:407] Connection header received
0808 18:34:30.940359 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.940362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.940413 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.940554 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.941732 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:30.941735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:30.941737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:34:30.941749 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:30.941912 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:34:30.941925 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:30.942025 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:30.942189 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 18:34:30.942232 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:34:30.943223 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:30.938121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34105 (local address 127.27.251.195:39789)
0808 18:34:30.938832 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:30.938874 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:30.940126 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:30.940776 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:30.940786 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:30.940888 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:30.941452 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:34:30.941453 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:30.941494 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:34:30.941571 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:30.942021 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:30.942039 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:30.942045 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 18:34:30.942201 (+ 156us) client_negotiation.cc:240] Negotiation successful
0808 18:34:30.942417 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260808 18:34:30.943284 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7223
I20260808 18:34:30.943398 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:30.943688 28655 external_mini_cluster.cc:1428] Setting key ce16b57af7378c757bc8159c702e167a
I20260808 18:34:30.944569 7357 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34105
I20260808 18:34:30.944829 7357 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:30.945250 7357 heartbeater.cc:511] Master 127.27.251.254:34105 requested a full tablet report, sending...
I20260808 18:34:30.946270 6882 ts_manager.cc:194] Registered new tserver with Master: 00f4b7f207454009b041dd1f6850cccb (127.27.251.195:37739)
I20260808 18:34:30.957715 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 18:34:30.960670 6923 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:34:30.959719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.27.251.254:34105)
0808 18:34:30.959876 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:30.959883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:30.960026 (+ 143us) server_negotiation.cc:407] Connection header received
0808 18:34:30.960139 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:30.960141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:30.960178 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:30.960248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:30.960458 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56330: BlockingRecv error: recv got EOF from 127.0.0.1:56330 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:34:30.960754 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6945
I20260808 18:34:30.969616 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7087
I20260808 18:34:30.977130 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7223
I20260808 18:34:30.983789 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 6849
2026-08-08T18:34:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (886 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:34:31 dist-test-slave-fd7l krb5kdc[7368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:34:31 dist-test-slave-fd7l krb5kdc[7368](info): set up 2 sockets
Aug 08 18:34:31 dist-test-slave-fd7l krb5kdc[7368](info): commencing operation
krb5kdc: starting...
W20260808 18:34:32.080715 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:34:32.211148 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41283
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:41283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:32.328550 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:32.328811 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:32.328851 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:32.332938 7381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:34:32.333014 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:32.333034 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:32.333051 7381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:34:32.333067 7381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:34:32.338595 7381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:41283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:32.339775 7381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:32.340929 7381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:32.349859 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:32.349864 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.349988 7381 server_base.cc:1034] running on GCE node
W20260808 18:34:32.350203 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.350768 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:32.351843 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:32.353032 7381 hybrid_clock.cc:648] HybridClock initialized: now 1786214072353010 us; error 31 us; skew 500 ppm
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:32.359467 7381 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:34:32.360373 7381 webserver.cc:460] Webserver started at http://127.27.251.254:46671/ using document root <none> and password file <none>
I20260808 18:34:32.360991 7381 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:32.361084 7381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:32.361326 7381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:32.363385 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "82c0c93c5872478bb54f36def86654c1"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "c1a34d644f4761bdf956448d75edfdc5"
server_key_iv: "1667c85c66919ba907a1faa8471c78ec"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.363960 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "82c0c93c5872478bb54f36def86654c1"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "c1a34d644f4761bdf956448d75edfdc5"
server_key_iv: "1667c85c66919ba907a1faa8471c78ec"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.367550 7381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:34:32.370088 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.371340 7381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:32.371480 7381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "82c0c93c5872478bb54f36def86654c1"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "c1a34d644f4761bdf956448d75edfdc5"
server_key_iv: "1667c85c66919ba907a1faa8471c78ec"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.371595 7381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:32.386974 7381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:32.390483 7381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:32.390707 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:32.398187 7381 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:41283
I20260808 18:34:32.398270 7451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:41283 every 8 connection(s)
I20260808 18:34:32.399228 7381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:34:32.402155 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:32.408011 7452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: Bootstrap starting.
I20260808 18:34:32.408651 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7381
I20260808 18:34:32.408790 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:34:32.409093 28655 external_mini_cluster.cc:1428] Setting key eb89674e656d4b97d37c6ea75fc7d7ef
I20260808 18:34:32.411111 7452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:32.411898 7452 log.cc:824] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:32.414391 7452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: No bootstrap required, opened a new log
I20260808 18:34:32.419135 7452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } }
I20260808 18:34:32.419399 7452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:32.419481 7452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c0c93c5872478bb54f36def86654c1, State: Initialized, Role: FOLLOWER
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:32.420007 7452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } }
I20260808 18:34:32.420164 7452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:34:32.420239 7452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:34:32.420382 7452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:32.421727 7452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } }
I20260808 18:34:32.422115 7452 leader_election.cc:302] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c0c93c5872478bb54f36def86654c1; no voters:
I20260808 18:34:32.422552 7452 leader_election.cc:288] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:34:32.422672 7457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:32.422902 7457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [term 1 LEADER]: Becoming Leader. State: Replica: 82c0c93c5872478bb54f36def86654c1, State: Running, Role: LEADER
I20260808 18:34:32.423327 7457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } }
I20260808 18:34:32.423975 7452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:34:32.425302 7458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c0c93c5872478bb54f36def86654c1. Latest consensus state: current_term: 1 leader_uuid: "82c0c93c5872478bb54f36def86654c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } } }
I20260808 18:34:32.425314 7459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c0c93c5872478bb54f36def86654c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c0c93c5872478bb54f36def86654c1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41283 } } }
I20260808 18:34:32.425473 7458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:32.425484 7459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1 [sys.catalog]: This master's current role is: LEADER
I20260808 18:34:32.425904 7468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:34:32.426828 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:32.410680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.27.251.254:41283)
0808 18:34:32.411195 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:32.411240 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:32.411296 (+ 56us) server_negotiation.cc:407] Connection header received
0808 18:34:32.412117 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:32.412139 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:32.412652 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:32.413074 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:32.414008 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.415233 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.416448 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.417069 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.417208 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:32.420365 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:32.420371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:32.420392 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:32.420428 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:32.422619 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.423198 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.423204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.423210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.423345 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.423701 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.423705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.423707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.424232 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:32.424464 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:32.425451 (+ 987us) server_negotiation.cc:299] Negotiation successful
0808 18:34:32.425804 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":181,"threads_started":1}
I20260808 18:34:32.429008 7468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:34:32.434692 7468 catalog_manager.cc:1347] Generated new cluster ID: e6539b60c36f4ed09d93ad4bf86f0b01
I20260808 18:34:32.434754 7468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:34:32.445283 7468 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:34:32.446175 7468 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:34:32.453576 7468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: Generated new TSK 0
I20260808 18:34:32.454469 7468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:34:32.489089 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:32.606426 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:32.606735 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:32.606812 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:32.610880 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:32.610962 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:32.611052 7479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:34:32.616390 7479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:32.617497 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:32.618592 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:32.629026 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.629225 7479 server_base.cc:1034] running on GCE node
W20260808 18:34:32.629015 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:32.629019 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.629801 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:32.630491 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:32.631649 7479 hybrid_clock.cc:648] HybridClock initialized: now 1786214072631617 us; error 50 us; skew 500 ppm
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:32.634600 7479 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:34:32.635494 7479 webserver.cc:460] Webserver started at http://127.27.251.193:33153/ using document root <none> and password file <none>
I20260808 18:34:32.636081 7479 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:32.636143 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:32.636376 7479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:32.638537 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "3208ddb8e2394a2c8f53d4547ea353b2"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "78ae5f9e71e735aa29d9a9003a8f05b0"
server_key_iv: "06ee139748ca0efc5e4fc604e820dbdc"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.639138 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "3208ddb8e2394a2c8f53d4547ea353b2"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "78ae5f9e71e735aa29d9a9003a8f05b0"
server_key_iv: "06ee139748ca0efc5e4fc604e820dbdc"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.643055 7479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:32.645550 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.646762 7479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:34:32.646899 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "3208ddb8e2394a2c8f53d4547ea353b2"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "78ae5f9e71e735aa29d9a9003a8f05b0"
server_key_iv: "06ee139748ca0efc5e4fc604e820dbdc"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.647010 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:32.675155 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:32.678592 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:32.678814 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:32.679451 7479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:32.680307 7479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:32.680385 7479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.680466 7479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:32.680510 7479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.690021 7479 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:38565
I20260808 18:34:32.690054 7616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:38565 every 8 connection(s)
I20260808 18:34:32.690944 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:34:32.696429 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7479
I20260808 18:34:32.696630 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:34:32.696976 28655 external_mini_cluster.cc:1428] Setting key 528475b45bcd1f8003f3832a10a52f9a
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:32.705680 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:32.692884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:52717 (local address 127.27.251.254:41283)
0808 18:34:32.693020 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:32.693024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:32.693906 (+ 882us) server_negotiation.cc:407] Connection header received
0808 18:34:32.695296 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:32.695299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:32.695347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:32.695426 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:32.696800 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.697535 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.698709 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.699134 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.699257 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:32.701857 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:32.701860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:32.701862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:32.701886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:32.703343 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.703980 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.703991 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.703993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.704072 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.704434 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.704439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.704441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.704697 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:32.704809 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:32.705357 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 18:34:32.705506 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:34:32.706656 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:32.693284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41283 (local address 127.27.251.193:52717)
0808 18:34:32.693742 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:32.693786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:32.695074 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:32.695591 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:32.695597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:32.695951 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:32.696609 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:32.696629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.697687 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.697690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:32.698585 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:32.698594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.699289 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.699292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:32.699440 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:32.700061 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:32.700087 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:32.701682 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:32.703465 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:32.703471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:32.703485 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:32.703865 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:32.704169 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:32.704172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:32.704174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:32.704333 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:32.704820 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:32.704829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:32.705111 (+ 282us) client_negotiation.cc:769] Sending connection context
0808 18:34:32.705317 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 18:34:32.705674 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:34:32.708132 7617 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41283
I20260808 18:34:32.708408 7617 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:32.708973 7617 heartbeater.cc:511] Master 127.27.251.254:41283 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:34:32.710731 7414 ts_manager.cc:194] Registered new tserver with Master: 3208ddb8e2394a2c8f53d4547ea353b2 (127.27.251.193:38565)
I20260808 18:34:32.711933 7414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:52717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:34:32.728201 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:32.846146 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:32.846416 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:32.846501 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:32.851423 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:32.851491 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:32.851600 7623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:34:32.856935 7623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:32.858026 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:32.859062 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:32.869243 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.869287 7623 server_base.cc:1034] running on GCE node
W20260808 18:34:32.869239 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:32.869239 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:32.869954 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:32.870592 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:32.871781 7623 hybrid_clock.cc:648] HybridClock initialized: now 1786214072871748 us; error 47 us; skew 500 ppm
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:32.874581 7623 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:34:32.875566 7623 webserver.cc:460] Webserver started at http://127.27.251.194:36247/ using document root <none> and password file <none>
I20260808 18:34:32.876217 7623 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:32.876312 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:32.876598 7623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:32.878736 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "154a72343c7c46cb677001cf39943c0e"
server_key_iv: "fca864006578b0b72aa64449338ef91b"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.879377 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "154a72343c7c46cb677001cf39943c0e"
server_key_iv: "fca864006578b0b72aa64449338ef91b"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.883384 7623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 18:34:32.885986 7638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.887269 7623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:34:32.887413 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8"
format_stamp: "Formatted at 2026-08-08 18:34:32 on dist-test-slave-fd7l"
server_key: "154a72343c7c46cb677001cf39943c0e"
server_key_iv: "fca864006578b0b72aa64449338ef91b"
server_key_version: "encryptionkey@0"
I20260808 18:34:32.887539 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:32.915514 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:32.918602 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:32.918828 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:32.919484 7623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:32.920398 7623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:32.920475 7623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.920589 7623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:32.920637 7623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:32.930325 7623 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:38709
I20260808 18:34:32.930352 7759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:38709 every 8 connection(s)
I20260808 18:34:32.931615 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:34:32.935600 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7623
I20260808 18:34:32.935737 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:34:32.936039 28655 external_mini_cluster.cc:1428] Setting key 3f60581e16566ce14d5a2be513be1624
Aug 08 18:34:32 dist-test-slave-fd7l krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:34:32.948141 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:32.933612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:50943 (local address 127.27.251.254:41283)
0808 18:34:32.933760 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:32.933764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:32.934668 (+ 904us) server_negotiation.cc:407] Connection header received
0808 18:34:32.936576 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:32.936584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:32.936678 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:32.936754 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:32.938342 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.938908 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.940432 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.940924 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.941066 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:32.943795 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:32.943801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:32.943808 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:32.943849 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:32.945752 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.946294 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.946300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.946306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.946415 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:32.946825 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:32.946831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:32.946835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:32.947119 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:32.947237 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:32.947775 (+ 538us) server_negotiation.cc:299] Negotiation successful
0808 18:34:32.947908 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:34:32.948976 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:32.934031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41283 (local address 127.27.251.194:50943)
0808 18:34:32.934508 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:32.934553 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:32.936190 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:32.936965 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:32.936973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:32.937444 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:32.938179 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:32.938204 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.939041 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.939047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:32.940300 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:32.940310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:32.941082 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:32.941085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:32.941253 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:32.941889 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:32.941906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:32.943594 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:32.945891 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:32.945896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:32.945910 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:32.946179 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:32.946525 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:32.946527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:32.946528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:32.946703 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:32.947250 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:32.947256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:32.947500 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:34:32.947672 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 18:34:32.947944 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260808 18:34:32.950485 7760 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41283
I20260808 18:34:32.950788 7760 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:34:32.951465 7760 heartbeater.cc:511] Master 127.27.251.254:41283 requested a full tablet report, sending...
I20260808 18:34:32.952755 7414 ts_manager.cc:194] Registered new tserver with Master: 5b89ea6d2a9e4bfc91e4babd7f86cfa8 (127.27.251.194:38709)
I20260808 18:34:32.953496 7414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:50943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:34:32.971385 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:34:33.088491 7766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:34:33.088842 7766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:34:33.088922 7766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:34:33.093254 7766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:34:33.093348 7766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:34:33.093441 7766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:34:33.098843 7766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:34:33.100056 7766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:34:33.101240 7766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:34:33.111640 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:34:33.111549 7772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:33.111738 7766 server_base.cc:1034] running on GCE node
W20260808 18:34:33.111550 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:34:33.112419 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:34:33.113128 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:34:33.114289 7766 hybrid_clock.cc:648] HybridClock initialized: now 1786214073114254 us; error 47 us; skew 500 ppm
Aug 08 18:34:33 dist-test-slave-fd7l krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:34:33.117256 7766 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:34:33.118261 7766 webserver.cc:460] Webserver started at http://127.27.251.195:36017/ using document root <none> and password file <none>
I20260808 18:34:33.118863 7766 fs_manager.cc:359] Metadata directory not provided
I20260808 18:34:33.118927 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:34:33.119182 7766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:34:33.121555 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "12709d5bdac84ea68e314dcd3e1dd9ea"
format_stamp: "Formatted at 2026-08-08 18:34:33 on dist-test-slave-fd7l"
server_key: "20081a417b88cf1e62272c1b8fd89b09"
server_key_iv: "3ebe73e1a29690a6c0356d0982694e82"
server_key_version: "encryptionkey@0"
I20260808 18:34:33.122192 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "12709d5bdac84ea68e314dcd3e1dd9ea"
format_stamp: "Formatted at 2026-08-08 18:34:33 on dist-test-slave-fd7l"
server_key: "20081a417b88cf1e62272c1b8fd89b09"
server_key_iv: "3ebe73e1a29690a6c0356d0982694e82"
server_key_version: "encryptionkey@0"
I20260808 18:34:33.126413 7766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:34:33.129118 7782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:33.130388 7766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:34:33.130514 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "12709d5bdac84ea68e314dcd3e1dd9ea"
format_stamp: "Formatted at 2026-08-08 18:34:33 on dist-test-slave-fd7l"
server_key: "20081a417b88cf1e62272c1b8fd89b09"
server_key_iv: "3ebe73e1a29690a6c0356d0982694e82"
server_key_version: "encryptionkey@0"
I20260808 18:34:33.130657 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:34:33.146451 7766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:34:33.149691 7766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:34:33.149911 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:34:33.150543 7766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:34:33.151443 7766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:34:33.151518 7766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:33.151592 7766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:34:33.151649 7766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:34:33.161012 7766 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:44639
I20260808 18:34:33.161089 7901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:44639 every 8 connection(s)
I20260808 18:34:33.161972 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:34:33.169806 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 7766
I20260808 18:34:33.169952 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:34:33.170331 28655 external_mini_cluster.cc:1428] Setting key 0a22306b51a2e534480d0631a5f2b123
Aug 08 18:34:33 dist-test-slave-fd7l krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:34:33.175797 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.163898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:47701 (local address 127.27.251.254:41283)
0808 18:34:33.164079 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.164084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.164939 (+ 855us) server_negotiation.cc:407] Connection header received
0808 18:34:33.166309 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.166312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.166361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.166439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:33.167601 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.168122 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.169293 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.169502 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.169595 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.172585 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:33.172588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:33.172591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:33.172618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:33.173922 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:33.174394 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:33.174397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:33.174398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:33.174471 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:33.174852 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:33.174854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:33.174856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:33.175027 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:33.175141 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.175564 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.175675 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 18:34:33.176776 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.164298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41283 (local address 127.27.251.195:47701)
0808 18:34:33.164790 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.164833 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.166110 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.166611 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.166618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.166995 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:34:33.167469 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.167482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.168231 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.168233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.169186 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.169196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.169671 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.169677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.169896 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.170579 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:34:33.170601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:34:33.172377 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:34:33.174032 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:33.174039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:33.174053 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:33.174306 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:33.174561 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:34:33.174564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:34:33.174565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:34:33.174755 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:34:33.175119 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:34:33.175124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:34:33.175351 (+ 227us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.175513 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.175892 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":156,"threads_started":1}
I20260808 18:34:33.177877 7902 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41283
I20260808 18:34:33.178138 7902 heartbeater.cc:464] Registering TS with master...
I20260808 18:34:33.178637 7902 heartbeater.cc:511] Master 127.27.251.254:41283 requested a full tablet report, sending...
I20260808 18:34:33.179643 7414 ts_manager.cc:194] Registered new tserver with Master: 12709d5bdac84ea68e314dcd3e1dd9ea (127.27.251.195:44639)
I20260808 18:34:33.180214 7414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:47701
I20260808 18:34:33.184362 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:34:33.194199 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.186391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.27.251.254:41283)
0808 18:34:33.186511 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.186515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.186703 (+ 188us) server_negotiation.cc:407] Connection header received
0808 18:34:33.186799 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.186801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.186839 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.186906 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:34:33.187582 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.188072 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.189084 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.189305 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.189385 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.191031 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:34:33.191034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:34:33.191036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:34:33.191057 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:34:33.192657 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:33.193073 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:33.193075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:33.193076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:33.193171 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:34:33.193415 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:34:33.193417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:34:33.193418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:34:33.193646 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:34:33.193750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.193957 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.194056 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:34:33.197454 7414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:34:33.200608 7414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:34:33.214671 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38709 (local address 127.0.0.1:54192)
0808 18:34:33.209730 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.209780 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.209915 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.210940 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.210942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.210980 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:33.211157 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.211165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212196 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.212201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.213398 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.213408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.214259 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.214262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.214338 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.214369 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.214458 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.214544 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":21,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20260808 18:34:33.214759 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.27.251.194:38709)
0808 18:34:33.210507 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.210517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.210537 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:34:33.210599 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.210603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.210806 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.210950 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:33.211301 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212060 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.213527 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.214139 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.214267 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.214397 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.214506 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.214596 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":116,"threads_started":1}
I20260808 18:34:33.215493 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:44639 (local address 127.0.0.1:52598)
0808 18:34:33.210584 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.210597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.210703 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.211113 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.211115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.211128 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:33.211208 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.211217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212670 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.212673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.213955 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.213966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.215077 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.215081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.215164 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.215187 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.215276 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.215359 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":59,"thread_start_us":103,"threads_started":1}
I20260808 18:34:33.215600 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.27.251.195:44639)
0808 18:34:33.210449 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.210460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.210708 (+ 248us) server_negotiation.cc:407] Connection header received
0808 18:34:33.210808 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.210813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.210982 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.211117 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:33.211305 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212534 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.214096 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.214949 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.215088 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.215215 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.215352 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.215456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":173,"threads_started":1}
I20260808 18:34:33.216922 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:38565 (local address 127.0.0.1:43576)
0808 18:34:33.210146 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.210159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.210244 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.210937 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.210942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.210981 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:33.211205 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.211214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212670 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.212673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.213954 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.213966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.216438 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.216442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.216597 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.216625 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.216714 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.216785 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":151,"spinlock_wait_cycles":6656,"thread_start_us":105,"threads_started":1}
I20260808 18:34:33.216976 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.209884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.27.251.193:38565)
0808 18:34:33.210248 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.210257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.210283 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:34:33.210365 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.210370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.210597 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.210756 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:33.211352 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.212495 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.215294 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.216119 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.216242 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.216373 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.216750 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.216840 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":69,"threads_started":1}
I20260808 18:34:33.218122 7833 tablet_service.cc:1467] Processing CreateTablet for tablet 245c86636df24cd5b5411489012a15f2 (DEFAULT_TABLE table=test-table [id=2eb4fc47cf2046e7b4d68714c385cd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:33.218125 7689 tablet_service.cc:1467] Processing CreateTablet for tablet 245c86636df24cd5b5411489012a15f2 (DEFAULT_TABLE table=test-table [id=2eb4fc47cf2046e7b4d68714c385cd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:33.219290 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245c86636df24cd5b5411489012a15f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:33.219292 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245c86636df24cd5b5411489012a15f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:33.219307 7547 tablet_service.cc:1467] Processing CreateTablet for tablet 245c86636df24cd5b5411489012a15f2 (DEFAULT_TABLE table=test-table [id=2eb4fc47cf2046e7b4d68714c385cd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:34:33.220415 7547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245c86636df24cd5b5411489012a15f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:34:33.226727 7922 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Bootstrap starting.
I20260808 18:34:33.227615 7921 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: Bootstrap starting.
I20260808 18:34:33.228893 7923 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Bootstrap starting.
I20260808 18:34:33.229450 7921 tablet_bootstrap.cc:654] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:33.229841 7922 tablet_bootstrap.cc:654] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:33.230392 7921 log.cc:824] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:33.230674 7922 log.cc:824] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:33.231549 7923 tablet_bootstrap.cc:654] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Neither blocks nor log segments found. Creating new log.
I20260808 18:34:33.232367 7923 log.cc:824] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Log is configured to *not* fsync() on all Append() calls
I20260808 18:34:33.232662 7922 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: No bootstrap required, opened a new log
I20260808 18:34:33.232740 7921 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: No bootstrap required, opened a new log
I20260808 18:34:33.232899 7922 ts_tablet_manager.cc:1397] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:34:33.233587 7921 ts_tablet_manager.cc:1397] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:34:33.237485 7922 raft_consensus.cc:348] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.237812 7922 raft_consensus.cc:374] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:33.237871 7922 raft_consensus.cc:729] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b89ea6d2a9e4bfc91e4babd7f86cfa8, State: Initialized, Role: FOLLOWER
I20260808 18:34:33.237905 7921 raft_consensus.cc:348] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.238159 7921 raft_consensus.cc:374] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:33.238221 7921 raft_consensus.cc:729] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3208ddb8e2394a2c8f53d4547ea353b2, State: Initialized, Role: FOLLOWER
I20260808 18:34:33.238384 7922 consensus_queue.cc:260] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.238567 7923 tablet_bootstrap.cc:492] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: No bootstrap required, opened a new log
I20260808 18:34:33.238678 7921 consensus_queue.cc:260] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.238768 7923 ts_tablet_manager.cc:1397] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 18:34:33.239204 7760 heartbeater.cc:503] Master 127.27.251.254:41283 was elected leader, sending a full tablet report...
I20260808 18:34:33.239499 7922 ts_tablet_manager.cc:1428] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:34:33.239643 7921 ts_tablet_manager.cc:1428] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:34:33.240023 7617 heartbeater.cc:503] Master 127.27.251.254:41283 was elected leader, sending a full tablet report...
I20260808 18:34:33.242756 7923 raft_consensus.cc:348] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.242990 7923 raft_consensus.cc:374] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:34:33.243057 7923 raft_consensus.cc:729] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12709d5bdac84ea68e314dcd3e1dd9ea, State: Initialized, Role: FOLLOWER
I20260808 18:34:33.243501 7923 consensus_queue.cc:260] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.244300 7902 heartbeater.cc:503] Master 127.27.251.254:41283 was elected leader, sending a full tablet report...
I20260808 18:34:33.244453 7923 ts_tablet_manager.cc:1428] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:34:33.249703 7929 raft_consensus.cc:482] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:34:33.249854 7929 raft_consensus.cc:504] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.251219 7929 leader_election.cc:288] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b89ea6d2a9e4bfc91e4babd7f86cfa8 (127.27.251.194:38709), 3208ddb8e2394a2c8f53d4547ea353b2 (127.27.251.193:38565)
I20260808 18:34:33.256224 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.251886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:52791 (local address 127.27.251.193:38565)
0808 18:34:33.252032 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.252036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.252052 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:34:33.252205 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.252208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.252250 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.252346 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:33.252622 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.253321 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.254835 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.255460 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.255638 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.255679 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.255998 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.256092 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:34:33.256248 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.251387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:38709 (local address 127.27.251.195:52445)
0808 18:34:33.251579 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.251596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.251707 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.252093 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.252096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.252111 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:33.252252 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.252257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.253276 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.253278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.254697 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.254709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.255620 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.255624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.255858 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.255898 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.255964 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.256077 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":141}
I20260808 18:34:33.256263 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.251613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:38565 (local address 127.27.251.195:52791)
0808 18:34:33.251958 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.251970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:34:33.252097 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:34:33.252356 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:34:33.252360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:34:33.252377 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:34:33.252475 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.252480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.253421 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.253424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.254697 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:34:33.254709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.255620 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.255624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:34:33.255814 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.255883 (+ 69us) client_negotiation.cc:769] Sending connection context
0808 18:34:33.255962 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 18:34:33.256084 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":176,"thread_start_us":118,"threads_started":1}
I20260808 18:34:33.256630 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.251657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:52445 (local address 127.27.251.194:38709)
0808 18:34:33.251804 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.251809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.251824 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:34:33.251876 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.251881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.251975 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.252086 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:34:33.252358 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.253161 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.254835 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.255460 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.255617 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.255672 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.256389 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.256478 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:34:33.257243 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245c86636df24cd5b5411489012a15f2" candidate_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" is_pre_election: true
I20260808 18:34:33.257217 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245c86636df24cd5b5411489012a15f2" candidate_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" is_pre_election: true
I20260808 18:34:33.257558 7709 raft_consensus.cc:2454] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12709d5bdac84ea68e314dcd3e1dd9ea in term 0.
I20260808 18:34:33.257558 7568 raft_consensus.cc:2454] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12709d5bdac84ea68e314dcd3e1dd9ea in term 0.
I20260808 18:34:33.258152 7786 leader_election.cc:302] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12709d5bdac84ea68e314dcd3e1dd9ea, 3208ddb8e2394a2c8f53d4547ea353b2, 5b89ea6d2a9e4bfc91e4babd7f86cfa8; no voters:
I20260808 18:34:33.258440 7929 raft_consensus.cc:2781] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:34:33.258530 7929 raft_consensus.cc:482] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:34:33.258575 7929 raft_consensus.cc:3037] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:33.259745 7929 raft_consensus.cc:504] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.260095 7929 leader_election.cc:288] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [CANDIDATE]: Term 1 election: Requested vote from peers 5b89ea6d2a9e4bfc91e4babd7f86cfa8 (127.27.251.194:38709), 3208ddb8e2394a2c8f53d4547ea353b2 (127.27.251.193:38565)
I20260808 18:34:33.260423 7709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245c86636df24cd5b5411489012a15f2" candidate_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8"
I20260808 18:34:33.260489 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245c86636df24cd5b5411489012a15f2" candidate_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3208ddb8e2394a2c8f53d4547ea353b2"
I20260808 18:34:33.260596 7709 raft_consensus.cc:3037] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:33.260694 7568 raft_consensus.cc:3037] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:34:33.261737 7709 raft_consensus.cc:2454] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12709d5bdac84ea68e314dcd3e1dd9ea in term 1.
I20260808 18:34:33.261749 7568 raft_consensus.cc:2454] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12709d5bdac84ea68e314dcd3e1dd9ea in term 1.
I20260808 18:34:33.262035 7785 leader_election.cc:302] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12709d5bdac84ea68e314dcd3e1dd9ea, 3208ddb8e2394a2c8f53d4547ea353b2; no voters:
I20260808 18:34:33.262264 7929 raft_consensus.cc:2781] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:34:33.262531 7929 raft_consensus.cc:686] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 1 LEADER]: Becoming Leader. State: Replica: 12709d5bdac84ea68e314dcd3e1dd9ea, State: Running, Role: LEADER
I20260808 18:34:33.262869 7929 consensus_queue.cc:237] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } }
I20260808 18:34:33.266173 7414 catalog_manager.cc:5495] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 12709d5bdac84ea68e314dcd3e1dd9ea (127.27.251.195). New cstate: current_term: 1 leader_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12709d5bdac84ea68e314dcd3e1dd9ea" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44639 } health_report { overall_health: HEALTHY } } }
I20260808 18:34:33.277544 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.273679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.27.251.195:44639)
0808 18:34:33.273824 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:34:33.273829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:34:33.273843 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:34:33.273965 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:34:33.273968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:34:33.274024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:34:33.274146 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:34:33.274409 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.274953 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.275930 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:34:33.276198 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:34:33.276307 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:34:33.276697 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:34:33.277138 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:34:33.277281 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:34:33.277325 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 18:34:33.277386 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:34:33.283587 7709 raft_consensus.cc:1261] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 1 FOLLOWER]: Refusing update from remote peer 12709d5bdac84ea68e314dcd3e1dd9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:33.283583 7568 raft_consensus.cc:1261] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 1 FOLLOWER]: Refusing update from remote peer 12709d5bdac84ea68e314dcd3e1dd9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:34:33.284343 7931 consensus_queue.cc:1035] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3208ddb8e2394a2c8f53d4547ea353b2" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:33.284545 7929 consensus_queue.cc:1035] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b89ea6d2a9e4bfc91e4babd7f86cfa8" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:34:33.300668 7413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:33.300855 7413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:34:33.303669 7413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82c0c93c5872478bb54f36def86654c1: Sending DeleteTablet for 3 replicas of tablet 245c86636df24cd5b5411489012a15f2
I20260808 18:34:33.304610 7833 tablet_service.cc:1514] Processing DeleteTablet for tablet 245c86636df24cd5b5411489012a15f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:33 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:52598
I20260808 18:34:33.304697 7547 tablet_service.cc:1514] Processing DeleteTablet for tablet 245c86636df24cd5b5411489012a15f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:33 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:43576
I20260808 18:34:33.305579 7946 tablet_replica.cc:330] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2: stopping tablet replica
I20260808 18:34:33.306123 7946 raft_consensus.cc:2229] T 245c86636df24cd5b5411489012a15f2 P 3208ddb8e2394a2c8f53d4547ea353b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:33.306166 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7479
I20260808 18:34:33.306285 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet 245c86636df24cd5b5411489012a15f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:34:33 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:54192
I20260808 18:34:33.314962 7947 tablet_replica.cc:330] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: stopping tablet replica
I20260808 18:34:33.315397 7947 raft_consensus.cc:2229] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:34:33.315824 7947 raft_consensus.cc:2258] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:33.317404 7947 ts_tablet_manager.cc:1905] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:34:33.318521 7945 tablet_replica.cc:330] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: stopping tablet replica
I20260808 18:34:33.318909 7945 raft_consensus.cc:2229] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:34:33.319736 7947 ts_tablet_manager.cc:1918] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:33.319823 7947 log.cc:1196] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/245c86636df24cd5b5411489012a15f2
I20260808 18:34:33.320073 7947 ts_tablet_manager.cc:1939] T 245c86636df24cd5b5411489012a15f2 P 5b89ea6d2a9e4bfc91e4babd7f86cfa8: Deleting consensus metadata
I20260808 18:34:33.320410 7945 raft_consensus.cc:2258] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:34:33.321321 7400 catalog_manager.cc:4841] TS 5b89ea6d2a9e4bfc91e4babd7f86cfa8 (127.27.251.194:38709): tablet 245c86636df24cd5b5411489012a15f2 (table test-table [id=2eb4fc47cf2046e7b4d68714c385cd98]) successfully deleted
I20260808 18:34:33.322614 7945 ts_tablet_manager.cc:1905] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:34:33.324345 7399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:38565 (error 108)
I20260808 18:34:33.325234 7945 ts_tablet_manager.cc:1918] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:34:33.325331 7945 log.cc:1196] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/245c86636df24cd5b5411489012a15f2
I20260808 18:34:33.325548 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:34:33.325163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:38565 (local address 127.0.0.1:57588)
0808 18:34:33.325326 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 18:34:33.325414 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 18:34:33.325601 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7623
I20260808 18:34:33.333642 7945 ts_tablet_manager.cc:1939] T 245c86636df24cd5b5411489012a15f2 P 12709d5bdac84ea68e314dcd3e1dd9ea: Deleting consensus metadata
W20260808 18:34:33.334174 7399 catalog_manager.cc:4584] TS 3208ddb8e2394a2c8f53d4547ea353b2 (127.27.251.193:38565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 245c86636df24cd5b5411489012a15f2: Network error: Client connection negotiation failed: client connection to 127.27.251.193:38565: connect: Connection refused (error 111)
I20260808 18:34:33.334980 7397 catalog_manager.cc:4841] TS 12709d5bdac84ea68e314dcd3e1dd9ea (127.27.251.195:44639): tablet 245c86636df24cd5b5411489012a15f2 (table test-table [id=2eb4fc47cf2046e7b4d68714c385cd98]) successfully deleted
I20260808 18:34:33.341678 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7766
I20260808 18:34:33.349423 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 7381
2026-08-08T18:34:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:34:33 dist-test-slave-fd7l krb5kdc[7951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:34:33 dist-test-slave-fd7l krb5kdc[7951](info): set up 2 sockets
Aug 08 18:34:33 dist-test-slave-fd7l krb5kdc[7951](info): commencing operation
krb5kdc: starting...
W20260808 18:34:34.443658 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:34:34 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:34:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangeradmin_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangeradmin_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangerlookup_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangerlookup_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
I20260808 18:34:34.629549 28655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5xHY3t/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/postgres -l logfile start
2026-08-08 18:34:37.156 UTC [7983] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 18:34:37.156 UTC [7983] LOG: listening on IPv4 address "127.27.251.212", port 48867
2026-08-08 18:34:37.160 UTC [7983] LOG: listening on Unix socket "/tmp/.s.PGSQL.48867"
2026-08-08 18:34:37.182 UTC [7986] LOG: database system was shut down at 2026-08-08 18:34:35 UTC
2026-08-08 18:34:37.187 UTC [7983] LOG: database system is ready to accept connections
I20260808 18:34:38.180557 28655 mini_postgres.cc:96] Postgres bound to 48867
2026-08-08 18:34:38.188 UTC [7994] FATAL: database "slave" does not exist
127.27.251.212:48867 - accepting connections
I20260808 18:34:38.189417 28655 mini_ranger.cc:161] Starting Ranger...
I20260808 18:34:38.206260 28655 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 18:34:38.697105 28655 mini_ranger.cc:87] Created ranger Postgres database
I20260808 18:34:38.697194 28655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin
2026-08-08 18:34:38,985 [I] DB FLAVOR :POSTGRES
2026-08-08 18:34:38,985 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:34:38,986 [I] Checking connection..
2026-08-08 18:34:38,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 18:34:39,287 [I] Checking connection passed.
2026-08-08 18:34:39,288 [I] --------- Verifying version history table ---------
2026-08-08 18:34:39,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:34:39,544 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:34:39,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:34:39,818 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:34:39,818 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 18:34:39,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 18:34:40.084 UTC [8063] WARNING: there is no transaction in progress
2026-08-08 18:34:40,091 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 18:34:40,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:34:40,345 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 18:34:40,346 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:34:40,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 18:34:40,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 18:34:40,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fd7l.c.gcp-upstream.internal','N') ;"
2026-08-08 18:34:41,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:34:41,382 [I] Table x_portal_user does not exist in database ranger
2026-08-08 18:34:41,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:34:41,644 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 18:34:41,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:34:41,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:34:42,204 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 18:34:42,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 18:34:43.172 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.188 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.204 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.220 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.238 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.259 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.319 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.326 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.343 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.352 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.362 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.376 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.386 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.392 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.398 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:43.405 UTC [8174] WARNING: there is no transaction in progress
2026-08-08 18:34:44,047 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 18:34:44,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 18:34:44,315 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 18:34:44,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:34:44,578 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 18:34:44,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:34:44,844 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 18:34:44,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:34:45,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:34:45,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fd7l.c.gcp-upstream.internal';"
2026-08-08 18:34:45,589 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 18:34:45,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 18:34:45,851 [I] DB_PATCHES have already been applied
I20260808 18:34:45.857257 28655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5xHY3t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 18:34:45.857339 28655 mini_ranger.cc:193] Using host: 127.27.251.212
I20260808 18:34:45.861825 28655 mini_ranger.cc:227] Ranger admin URL: http://127.27.251.212:51243
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:34:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 18:34:47.031678 28655 mini_ranger.cc:160] Time spent starting Ranger: real 8.842s user 0.005s sys 0.008s
Aug 08, 2026 6:35:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:35:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 6:35:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:35:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:35:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 18:35:09.807073 28655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangerkms_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/rangerkms_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.251.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.251.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 18:35:09 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214109, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 18:35:09.921110 28655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 18:35:09.934073 28655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 18:35:10.790009 28655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 18:35:10.790123 28655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms
I20260808 18:35:10.790262 28655 mini_ranger_kms.cc:227] Using postgres at 127.27.251.212:48867
2026-08-08 18:35:11,381 [I] DB FLAVOR :POSTGRES
2026-08-08 18:35:11,381 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:35:11,381 [I] Checking connection
2026-08-08 18:35:11,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 18:35:11,626 [I] connection success
2026-08-08 18:35:11,626 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 18:35:11,627 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 18:35:11,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 18:35:11,886 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 18:35:11,887 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:35:11,887 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 18:35:11,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:48867/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 18:35:12,144 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 18:35:12.617529 28655 mini_ranger_kms.cc:310] Created kms service
I20260808 18:35:12.825587 28655 mini_ranger_kms.cc:326] Created kudu user
I20260808 18:35:12.874399 28655 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 18:35:12 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214109, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:13.300585 28655 mini_ranger_kms.cc:384] Added ranger policy
I20260808 18:35:13.300752 28655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms:/tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 18:35:13.300778 28655 mini_ranger_kms.cc:240] Using host: 127.27.251.212
I20260808 18:35:13.306694 28655 mini_ranger_kms.cc:276] Ranger KMS PID: 8365
I20260808 18:35:13.306835 28655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.251.212:54547
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:35:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 18:35:15 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786214115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:15.656713 28655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.736s user 0.032s sys 0.267s
I20260808 18:35:15.656857 28655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 18:35:15.656977 28655 mini_ranger_kms.cc:401] 127.27.251.212:54547/kms/v1/keys
Aug 08 18:35:16 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
Aug 08 18:35:16 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
Aug 08 18:35:18 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:35:18.811712 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45247
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45247
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:54547/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:35:18.952739 8416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:35:18.953007 8416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:35:18.953048 8416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:35:18.957197 8416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:35:18.957257 8416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:35:18.957276 8416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:35:18.957293 8416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:35:18.957307 8416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:35:18.963021 8416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.251.212:54547/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45247
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:35:18.964115 8416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:35:18.965226 8416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:35:18.974150 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:18.974272 8416 server_base.cc:1034] running on GCE node
W20260808 18:35:18.974085 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:18.974104 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:18.975046 8416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:35:18.976270 8416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:35:18.977461 8416 hybrid_clock.cc:648] HybridClock initialized: now 1786214118977443 us; error 36 us; skew 500 ppm
Aug 08 18:35:18 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:18.984005 8416 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:35:18.985061 8416 webserver.cc:460] Webserver started at http://127.27.251.254:45787/ using document root <none> and password file <none>
I20260808 18:35:18.985641 8416 fs_manager.cc:359] Metadata directory not provided
I20260808 18:35:18.985724 8416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:35:18.985952 8416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:35:18 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:19.043650 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "c3f95a11ea2c47a9ae9d95db74557a0f"
format_stamp: "Formatted at 2026-08-08 18:35:19 on dist-test-slave-fd7l"
server_key: "7500720cb2ef73b4e9951280d37616d1"
server_key_iv: "cf1306b80f8aab923a38d17489944fbe"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:19.044301 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "c3f95a11ea2c47a9ae9d95db74557a0f"
format_stamp: "Formatted at 2026-08-08 18:35:19 on dist-test-slave-fd7l"
server_key: "7500720cb2ef73b4e9951280d37616d1"
server_key_iv: "cf1306b80f8aab923a38d17489944fbe"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:19.048496 8416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 18:35:19.063107 8432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:19.064819 8416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:35:19.064973 8416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "c3f95a11ea2c47a9ae9d95db74557a0f"
format_stamp: "Formatted at 2026-08-08 18:35:19 on dist-test-slave-fd7l"
server_key: "7500720cb2ef73b4e9951280d37616d1"
server_key_iv: "cf1306b80f8aab923a38d17489944fbe"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:19.065119 8416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:35:19.087890 8416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:35:19.090953 8416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:35:19.091183 8416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:35:19.099771 8416 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45247
I20260808 18:35:19.099830 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45247 every 8 connection(s)
I20260808 18:35:19.100811 8416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:35:19.103557 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 8416
I20260808 18:35:19.103699 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:35:19.103812 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:35:19.104033 28655 external_mini_cluster.cc:1428] Setting key 5f2a582698c5599ec3bf38aaf95c3cfb
I20260808 18:35:19.110914 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: Bootstrap starting.
I20260808 18:35:19.113627 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: Neither blocks nor log segments found. Creating new log.
I20260808 18:35:19.114646 8486 log.cc:824] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:35:19 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:35:19.117038 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: No bootstrap required, opened a new log
I20260808 18:35:19.120478 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } }
I20260808 18:35:19.120870 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:35:19.120975 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f95a11ea2c47a9ae9d95db74557a0f, State: Initialized, Role: FOLLOWER
I20260808 18:35:19.121564 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } }
I20260808 18:35:19.121737 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:35:19.121822 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:35:19.121965 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:35:19.122541 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:19.105551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.27.251.254:45247)
0808 18:35:19.106077 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:19.106094 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:19.106166 (+ 72us) server_negotiation.cc:407] Connection header received
0808 18:35:19.107312 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:19.107338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:19.107780 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:19.108184 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:35:19.109142 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:19.110750 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:19.111949 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:19.112707 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:19.112867 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:19.116044 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:35:19.116051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:35:19.116068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:35:19.116099 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:35:19.118950 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:19.119478 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:19.119484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:19.119491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:19.119631 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:19.119971 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:19.119974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:19.119976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:19.120571 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:35:19.120786 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:19.121188 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 18:35:19.121495 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260808 18:35:19.123031 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } }
I20260808 18:35:19.123389 8486 leader_election.cc:302] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f95a11ea2c47a9ae9d95db74557a0f; no voters:
I20260808 18:35:19.123720 8486 leader_election.cc:288] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:35:19.124033 8492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:35:19.124289 8492 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [term 1 LEADER]: Becoming Leader. State: Replica: c3f95a11ea2c47a9ae9d95db74557a0f, State: Running, Role: LEADER
I20260808 18:35:19.124667 8492 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } }
I20260808 18:35:19.125118 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:35:19.126183 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } } }
I20260808 18:35:19.126408 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [sys.catalog]: This master's current role is: LEADER
I20260808 18:35:19.126658 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f95a11ea2c47a9ae9d95db74557a0f. Latest consensus state: current_term: 1 leader_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f95a11ea2c47a9ae9d95db74557a0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45247 } } }
I20260808 18:35:19.126824 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f [sys.catalog]: This master's current role is: LEADER
I20260808 18:35:19.127168 8501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:35:19.127624 8486 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:35:19.133460 8501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:35:19.144409 8501 catalog_manager.cc:1347] Generated new cluster ID: e59a18c5fc144b3c84d30bde58b3098e
I20260808 18:35:19.144475 8501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:35:19.159497 8501 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:35:19.161104 8501 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:35:19.169781 8501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: Generated new TSK 0
I20260808 18:35:19.170799 8501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:35:19.824254 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:54547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:35:19.981381 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:35:19.981635 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:35:19.981678 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:35:19.987303 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:35:19.987385 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:35:19.987469 8533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:35:19.994812 8533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.251.212:54547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:35:19.995986 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:35:19.997277 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:35:20.010258 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:20.010277 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:20.010586 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:20.011011 8533 server_base.cc:1034] running on GCE node
I20260808 18:35:20.011529 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:35:20.012336 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:35:20.013537 8533 hybrid_clock.cc:648] HybridClock initialized: now 1786214120013507 us; error 45 us; skew 500 ppm
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:20.017155 8533 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:35:20.018085 8533 webserver.cc:460] Webserver started at http://127.27.251.193:39203/ using document root <none> and password file <none>
I20260808 18:35:20.018702 8533 fs_manager.cc:359] Metadata directory not provided
I20260808 18:35:20.018800 8533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:35:20.019034 8533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:20.038028 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "e82b12ac7ff045279f94865f85bb7958"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "e1d50d26c1c00e2192730e3e716b5eee"
server_key_iv: "8c83114d647e37f225e00b5c0eb3f8e2"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.038811 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "e82b12ac7ff045279f94865f85bb7958"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "e1d50d26c1c00e2192730e3e716b5eee"
server_key_iv: "8c83114d647e37f225e00b5c0eb3f8e2"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.057927 8533 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.006s sys 0.000s
I20260808 18:35:20.109026 8553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.116238 8533 fs_manager.cc:725] Time spent opening block manager: real 0.038s user 0.002s sys 0.001s
I20260808 18:35:20.116410 8533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "e82b12ac7ff045279f94865f85bb7958"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "e1d50d26c1c00e2192730e3e716b5eee"
server_key_iv: "8c83114d647e37f225e00b5c0eb3f8e2"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.116619 8533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:35:20.152099 8533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:35:20.156626 8533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:35:20.156898 8533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:35:20.157716 8533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:35:20.158936 8533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:35:20.159022 8533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.159102 8533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:35:20.159160 8533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:20.170976 8533 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:46229
I20260808 18:35:20.172212 8533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:35:20.176725 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 8533
I20260808 18:35:20.176872 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:35:20.177254 28655 external_mini_cluster.cc:1428] Setting key cbff270cebea240bb85924145b4174c4
I20260808 18:35:20.178087 8671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:46229 every 8 connection(s)
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:35:20.203030 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:20.178412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:45697 (local address 127.27.251.254:45247)
0808 18:35:20.183371 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:20.183378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:20.183395 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:35:20.183439 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:20.183445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:20.183503 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:20.183612 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:35:20.189137 (+ 5525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.190048 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.191844 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.192215 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.192300 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:20.198128 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:35:20.198134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:35:20.198138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:35:20.198171 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:35:20.200244 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:20.200965 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:20.200968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:20.200971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:20.201060 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:20.201447 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:20.201450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:20.201452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:20.201766 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:35:20.201859 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:20.202667 (+ 808us) server_negotiation.cc:299] Negotiation successful
0808 18:35:20.202807 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4858,"thread_start_us":95,"threads_started":1}
I20260808 18:35:20.205104 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:20.178910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45247 (local address 127.27.251.193:45697)
0808 18:35:20.181569 (+ 2659us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:20.181634 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:20.182910 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:20.183960 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:20.183970 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:20.184715 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:35:20.188931 (+ 4216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:20.188953 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.190214 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.190218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:20.191693 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:20.191705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.192564 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.192570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:20.192780 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:20.193586 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:35:20.193614 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:35:20.195543 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:35:20.200402 (+ 4859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:20.200409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:20.200430 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:20.200847 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:20.201164 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:20.201167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:20.201169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:20.201350 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:20.201933 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:35:20.201940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:35:20.202349 (+ 409us) client_negotiation.cc:769] Sending connection context
0808 18:35:20.203562 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0808 18:35:20.203944 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2396,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:35:20.212530 8672 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45247
I20260808 18:35:20.212841 8672 heartbeater.cc:464] Registering TS with master...
I20260808 18:35:20.213557 8672 heartbeater.cc:511] Master 127.27.251.254:45247 requested a full tablet report, sending...
I20260808 18:35:20.215512 8449 ts_manager.cc:194] Registered new tserver with Master: e82b12ac7ff045279f94865f85bb7958 (127.27.251.193:46229)
I20260808 18:35:20.217166 8449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:45697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:35:20.250486 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:54547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:35:20.422755 8678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:35:20.423234 8678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:35:20.423359 8678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:35:20.429756 8678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:35:20.429929 8678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:35:20.430078 8678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:35:20.437907 8678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.251.212:54547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:35:20.439615 8678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:35:20.441067 8678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:35:20.454792 8685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:20.455389 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:20.455710 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:20.457060 8678 server_base.cc:1034] running on GCE node
I20260808 18:35:20.457511 8678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:35:20.458304 8678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:35:20.461606 8678 hybrid_clock.cc:648] HybridClock initialized: now 1786214120461567 us; error 49 us; skew 500 ppm
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:20.465309 8678 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:35:20.466560 8678 webserver.cc:460] Webserver started at http://127.27.251.194:46057/ using document root <none> and password file <none>
I20260808 18:35:20.467268 8678 fs_manager.cc:359] Metadata directory not provided
I20260808 18:35:20.467414 8678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:35:20.467689 8678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:20.490392 8678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "8175c48d58684295be62212970a102c6"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "f9091a86c533dffe397f94c2279bb565"
server_key_iv: "ef3a17b228554a391c0170faf77efcaa"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.492609 8678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "8175c48d58684295be62212970a102c6"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "f9091a86c533dffe397f94c2279bb565"
server_key_iv: "ef3a17b228554a391c0170faf77efcaa"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.501770 8678 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.002s
I20260808 18:35:20.520857 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.522527 8678 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260808 18:35:20.522703 8678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "8175c48d58684295be62212970a102c6"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "f9091a86c533dffe397f94c2279bb565"
server_key_iv: "ef3a17b228554a391c0170faf77efcaa"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.522886 8678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:35:20.555236 8678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:35:20.559160 8678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:35:20.559419 8678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:35:20.560220 8678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:35:20.561379 8678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:35:20.561488 8678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.561594 8678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:35:20.561672 8678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.574836 8678 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:39483
I20260808 18:35:20.575011 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:39483 every 8 connection(s)
I20260808 18:35:20.575865 8678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:35:20.578652 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 8678
I20260808 18:35:20.578791 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:35:20.579128 28655 external_mini_cluster.cc:1428] Setting key d32330acef19f5d41355bee80db19f4f
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:35:20.601253 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:20.584714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:48333 (local address 127.27.251.254:45247)
0808 18:35:20.584839 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:20.584845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:20.587235 (+ 2390us) server_negotiation.cc:407] Connection header received
0808 18:35:20.588592 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:20.588595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:20.588661 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:20.588758 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:35:20.590380 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.591250 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.593017 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.593382 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.593461 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:20.596612 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:35:20.596616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:35:20.596619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:35:20.596658 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:35:20.598510 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:20.599207 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:20.599211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:20.599213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:20.599308 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:20.599771 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:20.599775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:20.599777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:20.600021 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:35:20.600113 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:20.600925 (+ 812us) server_negotiation.cc:299] Negotiation successful
0808 18:35:20.601074 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:35:20.603119 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:20.583616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45247 (local address 127.27.251.194:48333)
0808 18:35:20.587034 (+ 3418us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:20.587113 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:20.588295 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:20.589091 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:20.589101 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:20.589483 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:35:20.590196 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:20.590216 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.591421 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.591425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:20.592869 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:20.592881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:20.593691 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:20.593695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:20.593899 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:20.594623 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:35:20.594657 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:35:20.596385 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:35:20.598680 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:20.598687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:20.598705 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:20.599075 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:20.599429 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:20.599431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:20.599434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:20.599642 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:20.600198 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:35:20.600205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:35:20.600587 (+ 382us) client_negotiation.cc:769] Sending connection context
0808 18:35:20.601510 (+ 923us) client_negotiation.cc:240] Negotiation successful
0808 18:35:20.601894 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3169,"thread_start_us":132,"threads_started":1}
I20260808 18:35:20.604751 8815 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45247
I20260808 18:35:20.605016 8815 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:35:20.605682 8815 heartbeater.cc:511] Master 127.27.251.254:45247 requested a full tablet report, sending...
I20260808 18:35:20.606889 8449 ts_manager.cc:194] Registered new tserver with Master: 8175c48d58684295be62212970a102c6 (127.27.251.194:39483)
I20260808 18:35:20.607663 8449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:48333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:35:20.639120 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:54547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:36509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
W20260808 18:35:20.814998 8820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:35:20.815485 8820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:35:20.815603 8820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:35:20.821839 8820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:35:20.821996 8820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:35:20.822126 8820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:35:20.829798 8820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:36509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.251.212:54547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:35:20.831471 8820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:35:20.832844 8820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:35:20.863241 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:35:20.864224 8828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:20.864578 8820 server_base.cc:1034] running on GCE node
W20260808 18:35:20.864650 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:35:20.867779 8820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:35:20.868664 8820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:35:20.884130 8820 hybrid_clock.cc:648] HybridClock initialized: now 1786214120884025 us; error 142 us; skew 500 ppm
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:35:20.888190 8820 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:35:20.889599 8820 webserver.cc:460] Webserver started at http://127.27.251.195:41041/ using document root <none> and password file <none>
I20260808 18:35:20.890452 8820 fs_manager.cc:359] Metadata directory not provided
I20260808 18:35:20.890592 8820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:35:20.890900 8820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:35:20 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:20.912756 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "a8aac29be042438c9c456b6fdd02b153"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "d6daffad25d0e229ae798ea23a380cc2"
server_key_iv: "567b1dd82eab1d9fa540f718c423eac3"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.914660 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "a8aac29be042438c9c456b6fdd02b153"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "d6daffad25d0e229ae798ea23a380cc2"
server_key_iv: "567b1dd82eab1d9fa540f718c423eac3"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.920291 8820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:35:20.948037 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.950187 8820 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260808 18:35:20.950371 8820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "a8aac29be042438c9c456b6fdd02b153"
format_stamp: "Formatted at 2026-08-08 18:35:20 on dist-test-slave-fd7l"
server_key: "d6daffad25d0e229ae798ea23a380cc2"
server_key_iv: "567b1dd82eab1d9fa540f718c423eac3"
server_key_version: "kuduclusterkey@0"
I20260808 18:35:20.950562 8820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:35:20.984462 8820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:35:20.991457 8820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:35:20.991784 8820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:35:20.992717 8820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:35:20.994046 8820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:35:20.994171 8820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:20.994288 8820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:35:20.994374 8820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:35:21.008102 8820 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:32969
I20260808 18:35:21.009680 8820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:35:21.019920 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 8820
I20260808 18:35:21.020079 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:35:21.020437 28655 external_mini_cluster.cc:1428] Setting key fcf0d5870ffac8038453a488101226e8
I20260808 18:35:21.022027 8952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:32969 every 8 connection(s)
Aug 08 18:35:21 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:35:21.053457 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.029155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:50365 (local address 127.27.251.254:45247)
0808 18:35:21.029312 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.029317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.035628 (+ 6311us) server_negotiation.cc:407] Connection header received
0808 18:35:21.037335 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.037338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.037395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.037447 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:35:21.040979 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.041875 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.043881 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.044246 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.044337 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.048440 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:35:21.048444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:35:21.048447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:35:21.048481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:35:21.050677 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:21.051359 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:21.051363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:21.051365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:21.051467 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:21.051920 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:21.051937 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:21.051939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:21.052234 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:35:21.052341 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.053129 (+ 788us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.053280 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:35:21.055770 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.027799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45247 (local address 127.27.251.195:50365)
0808 18:35:21.035441 (+ 7642us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.035512 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.037062 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.037912 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.037923 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.038350 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:35:21.039105 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.039128 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.042082 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.042086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.043713 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.043727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.044614 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.044618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.044850 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.045653 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:35:21.045676 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:35:21.047988 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:35:21.050813 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:21.050819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:21.050838 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:21.051232 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:21.051594 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:35:21.051597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:35:21.051598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:35:21.051799 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:35:21.052415 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:35:21.052422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:35:21.052809 (+ 387us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.053964 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.054343 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7370,"thread_start_us":190,"threads_started":1}
I20260808 18:35:21.057446 8953 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45247
I20260808 18:35:21.057689 8953 heartbeater.cc:464] Registering TS with master...
I20260808 18:35:21.058286 8953 heartbeater.cc:511] Master 127.27.251.254:45247 requested a full tablet report, sending...
I20260808 18:35:21.059639 8449 ts_manager.cc:194] Registered new tserver with Master: a8aac29be042438c9c456b6fdd02b153 (127.27.251.195:32969)
I20260808 18:35:21.060570 8449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:50365
I20260808 18:35:21.064672 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:35:21.079109 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.27.251.254:45247)
0808 18:35:21.067624 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.067629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.068363 (+ 734us) server_negotiation.cc:407] Connection header received
0808 18:35:21.068621 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.068624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.068679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.068738 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:35:21.069883 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.070800 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.072322 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.072744 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.072833 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.074914 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:35:21.074918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:35:21.074922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:35:21.074967 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:35:21.076993 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:21.077556 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:21.077578 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:21.077580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:21.077679 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:35:21.078031 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:35:21.078034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:35:21.078036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:35:21.078270 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:35:21.078380 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.078773 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.078905 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:35:21.083525 8449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:35:21.088979 8449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:35:21.117533 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.105868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:32969 (local address 127.0.0.1:56570)
0808 18:35:21.108214 (+ 2346us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.108227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.108313 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.109907 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.109914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.109942 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:35:21.110198 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.110207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.111939 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.111943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.114515 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.114528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.116067 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.116071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.116210 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.116255 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.117224 (+ 969us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.117373 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"thread_start_us":124,"threads_started":1}
I20260808 18:35:21.118510 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.106078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.27.251.195:32969)
0808 18:35:21.106697 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.106713 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.108419 (+ 1706us) server_negotiation.cc:407] Connection header received
0808 18:35:21.108543 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.108549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.108796 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.108985 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:35:21.110425 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.111778 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.114696 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.115703 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.115838 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.115996 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.118235 (+ 2239us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.118351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":153,"threads_started":1}
I20260808 18:35:21.123770 8886 tablet_service.cc:1467] Processing CreateTablet for tablet b62cf321858d4a70bf470a77858e86ec (DEFAULT_TABLE table=test-table [id=32535442ec3743ad989ec21eb0338f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:35:21.125224 8886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62cf321858d4a70bf470a77858e86ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:35:21.131155 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.105156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:39483 (local address 127.0.0.1:37786)
0808 18:35:21.107916 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.107929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.108028 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.117410 (+ 9382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.117413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.117444 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:35:21.117532 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.117538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.121167 (+ 3629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.121170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.129604 (+ 8434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.129619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.130762 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.130766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.130862 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.130891 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.130940 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.131018 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2693,"thread_start_us":98,"threads_started":1}
I20260808 18:35:21.131875 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.105323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.27.251.194:39483)
0808 18:35:21.116722 (+ 11399us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.116740 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.116766 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:35:21.116857 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.116862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.117080 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.117254 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:35:21.117668 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.118765 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.129786 (+ 11021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.130615 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.131457 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.131600 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.131652 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.131741 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11279,"thread_start_us":78,"threads_started":1}
I20260808 18:35:21.134701 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.102892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:46229 (local address 127.0.0.1:52414)
0808 18:35:21.107470 (+ 4578us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.107528 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.107671 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.119534 (+ 11863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.119537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.119554 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:35:21.119652 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.119658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.132090 (+ 12432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.132093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.133214 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.133224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.134328 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.134332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.134420 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.134455 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.134506 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.134574 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4379,"thread_start_us":129,"threads_started":1}
I20260808 18:35:21.134908 8740 tablet_service.cc:1467] Processing CreateTablet for tablet b62cf321858d4a70bf470a77858e86ec (DEFAULT_TABLE table=test-table [id=32535442ec3743ad989ec21eb0338f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:35:21.135641 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.103002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.27.251.193:46229)
0808 18:35:21.119053 (+ 16051us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.119059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.119083 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:35:21.119157 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.119162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.119369 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.119802 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:35:21.119933 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.121023 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.133383 (+ 12360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.134184 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.135215 (+ 1031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.135347 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.135397 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.135508 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15928,"thread_start_us":71,"threads_started":1}
I20260808 18:35:21.136099 8740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62cf321858d4a70bf470a77858e86ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:35:21.139816 8605 tablet_service.cc:1467] Processing CreateTablet for tablet b62cf321858d4a70bf470a77858e86ec (DEFAULT_TABLE table=test-table [id=32535442ec3743ad989ec21eb0338f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:35:21.140903 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b62cf321858d4a70bf470a77858e86ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:35:21.157814 8973 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Bootstrap starting.
I20260808 18:35:21.161507 8975 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: Bootstrap starting.
I20260808 18:35:21.161722 8973 tablet_bootstrap.cc:654] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Neither blocks nor log segments found. Creating new log.
I20260808 18:35:21.162667 8973 log.cc:824] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Log is configured to *not* fsync() on all Append() calls
I20260808 18:35:21.164997 8974 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Bootstrap starting.
I20260808 18:35:21.171617 8974 tablet_bootstrap.cc:654] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Neither blocks nor log segments found. Creating new log.
I20260808 18:35:21.166900 8975 tablet_bootstrap.cc:654] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: Neither blocks nor log segments found. Creating new log.
I20260808 18:35:21.178020 8975 log.cc:824] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: Log is configured to *not* fsync() on all Append() calls
I20260808 18:35:21.179364 8974 log.cc:824] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Log is configured to *not* fsync() on all Append() calls
I20260808 18:35:21.184793 8974 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: No bootstrap required, opened a new log
I20260808 18:35:21.184793 8973 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: No bootstrap required, opened a new log
I20260808 18:35:21.185040 8973 ts_tablet_manager.cc:1397] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260808 18:35:21.186977 8974 ts_tablet_manager.cc:1397] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260808 18:35:21.189383 8973 raft_consensus.cc:348] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.189756 8973 raft_consensus.cc:374] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:35:21.189821 8973 raft_consensus.cc:729] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8aac29be042438c9c456b6fdd02b153, State: Initialized, Role: FOLLOWER
I20260808 18:35:21.190354 8973 consensus_queue.cc:260] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.191390 8973 ts_tablet_manager.cc:1428] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:35:21.192200 8953 heartbeater.cc:503] Master 127.27.251.254:45247 was elected leader, sending a full tablet report...
I20260808 18:35:21.198019 8975 tablet_bootstrap.cc:492] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: No bootstrap required, opened a new log
I20260808 18:35:21.198253 8975 ts_tablet_manager.cc:1397] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20260808 18:35:21.199838 8974 raft_consensus.cc:348] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.200384 8974 raft_consensus.cc:374] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:35:21.200531 8974 raft_consensus.cc:729] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8175c48d58684295be62212970a102c6, State: Initialized, Role: FOLLOWER
I20260808 18:35:21.201174 8974 consensus_queue.cc:260] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.202171 8975 raft_consensus.cc:348] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.202443 8975 raft_consensus.cc:374] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:35:21.202498 8975 raft_consensus.cc:729] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82b12ac7ff045279f94865f85bb7958, State: Initialized, Role: FOLLOWER
I20260808 18:35:21.202999 8975 consensus_queue.cc:260] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.203920 8975 ts_tablet_manager.cc:1428] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:35:21.208703 8672 heartbeater.cc:503] Master 127.27.251.254:45247 was elected leader, sending a full tablet report...
I20260808 18:35:21.202394 8974 ts_tablet_manager.cc:1428] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Time spent starting tablet: real 0.015s user 0.000s sys 0.006s
I20260808 18:35:21.209923 8815 heartbeater.cc:503] Master 127.27.251.254:45247 was elected leader, sending a full tablet report...
Aug 08 18:35:21 dist-test-slave-fd7l krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:35:21.274557 8980 raft_consensus.cc:482] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:35:21.274809 8980 raft_consensus.cc:504] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.276337 8980 leader_election.cc:288] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82b12ac7ff045279f94865f85bb7958 (127.27.251.193:46229), a8aac29be042438c9c456b6fdd02b153 (127.27.251.195:32969)
W20260808 18:35:21.276376 8954 tablet.cc:2369] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:35:21.283736 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.276750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:58011 (local address 127.27.251.195:32969)
0808 18:35:21.276908 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.276913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.278096 (+ 1183us) server_negotiation.cc:407] Connection header received
0808 18:35:21.278270 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.278274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.278331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.278417 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:35:21.279906 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.280683 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.282217 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.282792 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.282920 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.282976 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.283499 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.283591 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:35:21.284564 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.276684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:32969 (local address 127.27.251.194:58011)
0808 18:35:21.277974 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.277992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.278137 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.278584 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.278587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.278611 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:35:21.278764 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.278771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.280808 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.280812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.282058 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.282071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.282971 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.282978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.283320 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.283350 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.284283 (+ 933us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.284356 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":140,"threads_started":1}
I20260808 18:35:21.285310 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cf321858d4a70bf470a77858e86ec" candidate_uuid: "8175c48d58684295be62212970a102c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8aac29be042438c9c456b6fdd02b153" is_pre_election: true
I20260808 18:35:21.285492 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.276684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:46229 (local address 127.27.251.194:53251)
0808 18:35:21.278396 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.278412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:35:21.278585 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:35:21.279055 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:35:21.279058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:35:21.279079 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:35:21.279227 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.279233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.280246 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.280250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.283999 (+ 3749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:35:21.284010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.285031 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.285035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:35:21.285137 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.285163 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:35:21.285247 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:35:21.285324 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":15,"thread_start_us":124,"threads_started":1}
I20260808 18:35:21.285509 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.277153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:53251 (local address 127.27.251.193:46229)
0808 18:35:21.277269 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.277273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.278496 (+ 1223us) server_negotiation.cc:407] Connection header received
0808 18:35:21.278735 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.278738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.278810 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.278872 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:35:21.279365 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.280100 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.284131 (+ 4031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.284773 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.284898 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.284949 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.285286 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.285368 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:35:21.285738 8906 raft_consensus.cc:2454] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8175c48d58684295be62212970a102c6 in term 0.
I20260808 18:35:21.286180 8613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cf321858d4a70bf470a77858e86ec" candidate_uuid: "8175c48d58684295be62212970a102c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82b12ac7ff045279f94865f85bb7958" is_pre_election: true
I20260808 18:35:21.286356 8698 leader_election.cc:302] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8175c48d58684295be62212970a102c6, a8aac29be042438c9c456b6fdd02b153; no voters:
I20260808 18:35:21.286520 8613 raft_consensus.cc:2454] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8175c48d58684295be62212970a102c6 in term 0.
I20260808 18:35:21.286723 8980 raft_consensus.cc:2781] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:35:21.286819 8980 raft_consensus.cc:482] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:35:21.286871 8980 raft_consensus.cc:3037] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:35:21.288086 8980 raft_consensus.cc:504] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.288492 8980 leader_election.cc:288] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [CANDIDATE]: Term 1 election: Requested vote from peers e82b12ac7ff045279f94865f85bb7958 (127.27.251.193:46229), a8aac29be042438c9c456b6fdd02b153 (127.27.251.195:32969)
I20260808 18:35:21.288844 8613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cf321858d4a70bf470a77858e86ec" candidate_uuid: "8175c48d58684295be62212970a102c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82b12ac7ff045279f94865f85bb7958"
I20260808 18:35:21.289013 8906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b62cf321858d4a70bf470a77858e86ec" candidate_uuid: "8175c48d58684295be62212970a102c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8aac29be042438c9c456b6fdd02b153"
I20260808 18:35:21.289012 8613 raft_consensus.cc:3037] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:35:21.289146 8906 raft_consensus.cc:3037] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:35:21.290501 8613 raft_consensus.cc:2454] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8175c48d58684295be62212970a102c6 in term 1.
I20260808 18:35:21.290501 8906 raft_consensus.cc:2454] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8175c48d58684295be62212970a102c6 in term 1.
I20260808 18:35:21.290835 8696 leader_election.cc:302] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8175c48d58684295be62212970a102c6, e82b12ac7ff045279f94865f85bb7958; no voters:
I20260808 18:35:21.291038 8980 raft_consensus.cc:2781] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:35:21.291342 8980 raft_consensus.cc:686] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8175c48d58684295be62212970a102c6, State: Running, Role: LEADER
I20260808 18:35:21.291756 8980 consensus_queue.cc:237] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } }
I20260808 18:35:21.295492 8449 catalog_manager.cc:5495] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8175c48d58684295be62212970a102c6 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "8175c48d58684295be62212970a102c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8175c48d58684295be62212970a102c6" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 } health_report { overall_health: UNKNOWN } } }
I20260808 18:35:21.319429 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.314492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.27.251.194:39483)
0808 18:35:21.314645 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:35:21.314650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:35:21.314666 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:35:21.314754 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:35:21.314757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:35:21.314828 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:35:21.314948 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:35:21.315259 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.316060 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.317289 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:35:21.317709 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:35:21.317844 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:35:21.318186 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:35:21.318769 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:35:21.318888 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:35:21.319224 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 18:35:21.319285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:35:21.328605 8906 raft_consensus.cc:1261] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 1 FOLLOWER]: Refusing update from remote peer 8175c48d58684295be62212970a102c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:35:21.329391 8980 consensus_queue.cc:1035] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8aac29be042438c9c456b6fdd02b153" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:35:21.333135 8613 raft_consensus.cc:1261] T b62cf321858d4a70bf470a77858e86ec P e82b12ac7ff045279f94865f85bb7958 [term 1 FOLLOWER]: Refusing update from remote peer 8175c48d58684295be62212970a102c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:35:21.334350 8980 consensus_queue.cc:1035] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82b12ac7ff045279f94865f85bb7958" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:35:21.335812 8816 tablet.cc:2369] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:35:21.345479 8990 mvcc.cc:205] Tried to move back new op lower bound from 7316333040952332288 to 7316333040814862336. Current Snapshot: MvccSnapshot[applied={T|T < 7316333040952332288}]
I20260808 18:35:21.346684 8993 mvcc.cc:205] Tried to move back new op lower bound from 7316333040952332288 to 7316333040814862336. Current Snapshot: MvccSnapshot[applied={T|T < 7316333040952332288 or (T in {7316333040952332288})}]
I20260808 18:35:21.363710 8447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:35:21.363917 8447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:35:21.374891 8447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f95a11ea2c47a9ae9d95db74557a0f: Sending DeleteTablet for 3 replicas of tablet b62cf321858d4a70bf470a77858e86ec
I20260808 18:35:21.376225 8605 tablet_service.cc:1514] Processing DeleteTablet for tablet b62cf321858d4a70bf470a77858e86ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:35:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:52414
I20260808 18:35:21.376256 8886 tablet_service.cc:1514] Processing DeleteTablet for tablet b62cf321858d4a70bf470a77858e86ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:35:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:56570
I20260808 18:35:21.376844 8740 tablet_service.cc:1514] Processing DeleteTablet for tablet b62cf321858d4a70bf470a77858e86ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:35:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37786
I20260808 18:35:21.377126 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 8533
I20260808 18:35:21.388350 9000 tablet_replica.cc:330] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: stopping tablet replica
I20260808 18:35:21.388844 9000 raft_consensus.cc:2229] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:35:21.389282 9000 raft_consensus.cc:2258] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:35:21.400820 9002 tablet_replica.cc:330] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: stopping tablet replica
I20260808 18:35:21.401278 9002 raft_consensus.cc:2229] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:35:21.401813 9002 raft_consensus.cc:2258] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:35:21.402477 9000 ts_tablet_manager.cc:1905] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:35:21.403481 9002 ts_tablet_manager.cc:1905] T b62cf321858d4a70bf470a77858e86ec P 8175c48d58684295be62212970a102c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:35:21.405516 9000 ts_tablet_manager.cc:1918] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:35:21.405612 9000 log.cc:1196] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/b62cf321858d4a70bf470a77858e86ec
I20260808 18:35:21.405975 9000 ts_tablet_manager.cc:1939] T b62cf321858d4a70bf470a77858e86ec P a8aac29be042438c9c456b6fdd02b153: Deleting consensus metadata
W20260808 18:35:21.407214 8433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:46229 (error 108)
I20260808 18:35:21.409510 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 8678
I20260808 18:35:21.409662 8435 catalog_manager.cc:4841] TS a8aac29be042438c9c456b6fdd02b153 (127.27.251.195:32969): tablet b62cf321858d4a70bf470a77858e86ec (table test-table [id=32535442ec3743ad989ec21eb0338f6b]) successfully deleted
I20260808 18:35:21.412132 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.411753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:46229 (local address 127.0.0.1:52424)
0808 18:35:21.411926 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.412023 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 18:35:21.412340 8433 catalog_manager.cc:4584] TS e82b12ac7ff045279f94865f85bb7958 (127.27.251.193:46229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b62cf321858d4a70bf470a77858e86ec: Network error: Client connection negotiation failed: client connection to 127.27.251.193:46229: connect: Connection refused (error 111)
I20260808 18:35:21.421386 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 8820
I20260808 18:35:21.422600 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:35:21.421606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:39483 (local address 127.0.0.1:37806)
0808 18:35:21.422414 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 18:35:21.422484 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.194:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693}
I20260808 18:35:21.432796 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 8416
I20260808 18:35:21.443087 28655 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 18:35:21.797351 28655 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 18:35:21.797526 28655 mini_ranger.cc:66] Stopping Ranger...
I20260808 18:35:22.152393 28655 mini_ranger.cc:68] Stopped Ranger
2026-08-08 18:35:22.152 UTC [7983] LOG: received smart shutdown request
2026-08-08 18:35:22.157 UTC [7983] LOG: background worker "logical replication launcher" (PID 7992) exited with exit code 1
2026-08-08 18:35:22.158 UTC [7987] LOG: shutting down
2026-08-08 18:35:22.349 UTC [7983] LOG: database system is shut down
2026-08-08T18:35:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49139 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:35:22 dist-test-slave-fd7l krb5kdc[9012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:35:22 dist-test-slave-fd7l krb5kdc[9012](info): set up 2 sockets
Aug 08 18:35:22 dist-test-slave-fd7l krb5kdc[9012](info): commencing operation
krb5kdc: starting...
W20260808 18:35:23.571094 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:35:23 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:35:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangeradmin_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangeradmin_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangerlookup_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangerlookup_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
I20260808 18:35:23.751673 28655 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5xHY3t/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/postgres -l logfile start
2026-08-08 18:35:26.363 UTC [9042] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 18:35:26.363 UTC [9042] LOG: listening on IPv4 address "127.27.251.212", port 42815
2026-08-08 18:35:26.367 UTC [9042] LOG: listening on Unix socket "/tmp/.s.PGSQL.42815"
2026-08-08 18:35:26.407 UTC [9045] LOG: database system was shut down at 2026-08-08 18:35:24 UTC
2026-08-08 18:35:26.411 UTC [9042] LOG: database system is ready to accept connections
I20260808 18:35:27.522908 28655 mini_postgres.cc:96] Postgres bound to 42815
2026-08-08 18:35:27.530 UTC [9053] FATAL: database "slave" does not exist
127.27.251.212:42815 - accepting connections
I20260808 18:35:27.531352 28655 mini_ranger.cc:161] Starting Ranger...
I20260808 18:35:27.549121 28655 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 18:35:28.065863 28655 mini_ranger.cc:87] Created ranger Postgres database
I20260808 18:35:28.065959 28655 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin
2026-08-08 18:35:28,361 [I] DB FLAVOR :POSTGRES
2026-08-08 18:35:28,361 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:35:28,361 [I] Checking connection..
2026-08-08 18:35:28,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 18:35:28,633 [I] Checking connection passed.
2026-08-08 18:35:28,633 [I] --------- Verifying version history table ---------
2026-08-08 18:35:28,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:35:28,901 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:35:28,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:35:29,136 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 18:35:29,136 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 18:35:29,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 18:35:29.401 UTC [9121] WARNING: there is no transaction in progress
2026-08-08 18:35:29,409 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 18:35:29,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 18:35:29,665 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 18:35:29,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:35:29,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 18:35:29,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 18:35:30,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fd7l.c.gcp-upstream.internal','N') ;"
2026-08-08 18:35:30,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:35:30,699 [I] Table x_portal_user does not exist in database ranger
2026-08-08 18:35:30,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:35:30,967 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 18:35:30,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:35:31,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:35:31,536 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 18:35:31,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 18:35:32.523 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.542 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.560 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.577 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.594 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.613 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.668 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.675 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.689 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.700 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.711 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.725 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.734 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.741 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.748 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:32.755 UTC [9236] WARNING: there is no transaction in progress
2026-08-08 18:35:33,441 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 18:35:33,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 18:35:33,695 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 18:35:33,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 18:35:33,964 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 18:35:33,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 18:35:34,237 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 18:35:34,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 18:35:34,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 18:35:34,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fd7l.c.gcp-upstream.internal';"
2026-08-08 18:35:35,009 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 18:35:35,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 18:35:35,274 [I] DB_PATCHES have already been applied
I20260808 18:35:35.280395 28655 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5xHY3t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 18:35:35.280467 28655 mini_ranger.cc:193] Using host: 127.27.251.212
I20260808 18:35:35.285218 28655 mini_ranger.cc:227] Ranger admin URL: http://127.27.251.212:41421
Aug 08, 2026 6:35:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:35:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:35:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 6:35:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 6:35:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 6:35:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 6:35:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 18:35:36.471612 28655 mini_ranger.cc:160] Time spent starting Ranger: real 8.940s user 0.005s sys 0.007s
Aug 08, 2026 6:35:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 6:35:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 6:35:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 6:35:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 6:35:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 18:35:59.431578 28655 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.251.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.251.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangerkms_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.251.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/rangerkms_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.251.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.251.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.251.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/HTTP_127.27.251.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 18:35:59 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214159, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 18:35:59.552947 28655 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 18:35:59.564338 28655 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 18:36:00.553027 28655 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 18:36:00.553126 28655 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms
I20260808 18:36:00.553151 28655 mini_ranger_kms.cc:227] Using postgres at 127.27.251.212:42815
2026-08-08 18:36:00,745 [I] DB FLAVOR :POSTGRES
2026-08-08 18:36:00,745 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 18:36:00,746 [I] Checking connection
2026-08-08 18:36:00,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 18:36:01,011 [I] connection success
2026-08-08 18:36:01,011 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 18:36:01,012 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 18:36:01,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 18:36:01,246 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 18:36:01,246 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 18:36:01,247 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 18:36:01,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.251.212:42815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 18:36:01,544 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 18:36:02.025864 28655 mini_ranger_kms.cc:310] Created kms service
I20260808 18:36:02.275245 28655 mini_ranger_kms.cc:326] Created kudu user
I20260808 18:36:02.328935 28655 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 18:36:02 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214159, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:02.800277 28655 mini_ranger_kms.cc:384] Added ranger policy
I20260808 18:36:02.800438 28655 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms:/tmp/dist-test-task5xHY3t/build/debug/bin/postgresql.jar:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5xHY3t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 18:36:02.800462 28655 mini_ranger_kms.cc:240] Using host: 127.27.251.212
I20260808 18:36:02.805090 28655 mini_ranger_kms.cc:276] Ranger KMS PID: 9424
I20260808 18:36:02.805190 28655 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.251.212:51031
Aug 08, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 6:36:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 6:36:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 6:36:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 18:36:05.114431 28655 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.561s user 0.025s sys 0.151s
I20260808 18:36:05.114593 28655 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 18:36:05.114676 28655 mini_ranger_kms.cc:401] 127.27.251.212:51031/kms/v1/keys
Aug 08 18:36:05 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786214165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 18:36:05 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
Aug 08 18:36:06 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.251.212@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
Aug 08 18:36:08 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:36:08.320767 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45563
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45563
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:08.457883 9476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:08.458196 9476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:08.458276 9476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:08.462328 9476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:08.462411 9476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:08.462446 9476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:08.462494 9476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:08.462553 9476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:08.468034 9476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.251.212:51031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45563
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:08.469249 9476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:08.470305 9476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:08.479439 9483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:08.479446 9482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:08.479553 9485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:08.479683 9476 server_base.cc:1034] running on GCE node
I20260808 18:36:08.480429 9476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:08.481642 9476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:08.482852 9476 hybrid_clock.cc:648] HybridClock initialized: now 1786214168482834 us; error 32 us; skew 500 ppm
Aug 08 18:36:08 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:08.497409 9476 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:08.498536 9476 webserver.cc:460] Webserver started at http://127.27.251.254:34259/ using document root <none> and password file <none>
I20260808 18:36:08.499151 9476 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:08.499209 9476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:08.499388 9476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:36:08 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:08.552428 9476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "a0e158690fe04c879e893be751147b78"
format_stamp: "Formatted at 2026-08-08 18:36:08 on dist-test-slave-fd7l"
server_key: "8f4b3b06ef231f1a69228ed0f82195a8"
server_key_iv: "64629c1cd0bea6e9300a5c38bed9d6d1"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:08.553158 9476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "a0e158690fe04c879e893be751147b78"
format_stamp: "Formatted at 2026-08-08 18:36:08 on dist-test-slave-fd7l"
server_key: "8f4b3b06ef231f1a69228ed0f82195a8"
server_key_iv: "64629c1cd0bea6e9300a5c38bed9d6d1"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:08.557452 9476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:36:08.572417 9492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:08.573961 9476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:08.574123 9476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a0e158690fe04c879e893be751147b78"
format_stamp: "Formatted at 2026-08-08 18:36:08 on dist-test-slave-fd7l"
server_key: "8f4b3b06ef231f1a69228ed0f82195a8"
server_key_iv: "64629c1cd0bea6e9300a5c38bed9d6d1"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:08.574275 9476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:08.607916 9476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:08.611177 9476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:08.611405 9476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:08.620051 9476 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45563
I20260808 18:36:08.620069 9546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45563 every 8 connection(s)
I20260808 18:36:08.621201 9476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:08.621863 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 9476
I20260808 18:36:08.622020 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:36:08.622329 28655 external_mini_cluster.cc:1428] Setting key a561112cc50935304308a4fad20bbf82
I20260808 18:36:08.626089 9547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 18:36:08 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:08.633709 9547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Bootstrap starting.
I20260808 18:36:08.636623 9547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:08.637133 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:08.623910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.27.251.254:45563)
0808 18:36:08.624362 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:08.624379 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:08.624416 (+ 37us) server_negotiation.cc:407] Connection header received
0808 18:36:08.625171 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:08.625187 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:08.625532 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:08.625888 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:08.626779 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:08.627794 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:08.628761 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:08.629238 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:08.629413 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:08.632017 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:08.632023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:08.632035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:08.632065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:08.633773 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:08.634260 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:08.634266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:08.634272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:08.634395 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:08.634689 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:08.634694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:08.634696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:08.635103 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:08.635325 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:08.635627 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 18:36:08.635944 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260808 18:36:08.637580 9547 log.cc:824] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:08.640816 9547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: No bootstrap required, opened a new log
I20260808 18:36:08.644694 9547 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:08.644919 9547 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:08.644955 9547 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e158690fe04c879e893be751147b78, State: Initialized, Role: FOLLOWER
I20260808 18:36:08.645315 9547 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:08.645424 9547 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:08.645496 9547 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:08.645601 9547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:08.646519 9547 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:08.646879 9547 leader_election.cc:302] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e158690fe04c879e893be751147b78; no voters:
I20260808 18:36:08.647208 9547 leader_election.cc:288] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:36:08.647356 9552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:08.647576 9552 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 LEADER]: Becoming Leader. State: Replica: a0e158690fe04c879e893be751147b78, State: Running, Role: LEADER
I20260808 18:36:08.647917 9552 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:08.648558 9547 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:08.649842 9553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e158690fe04c879e893be751147b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } } }
I20260808 18:36:08.649838 9554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e158690fe04c879e893be751147b78. Latest consensus state: current_term: 1 leader_uuid: "a0e158690fe04c879e893be751147b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } } }
I20260808 18:36:08.649977 9553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:08.649986 9554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:08.650481 9561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:08.653301 9561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:08.653738 9547 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:36:08.663156 9561 catalog_manager.cc:1347] Generated new cluster ID: 862f717c42fd4b6294becb7101b7f6c7
I20260808 18:36:08.663235 9561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:08.671949 9561 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:36:08.673216 9561 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:08.683066 9561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Generated new TSK 0
I20260808 18:36:08.683893 9561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:36:09.436455 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:09.597283 9592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:09.597636 9592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:09.597756 9592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:09.603569 9592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:09.603655 9592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:09.603776 9592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:09.610445 9592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:09.611965 9592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:09.613377 9592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:09.631564 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:09.634536 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:09.634706 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:09.636127 9592 server_base.cc:1034] running on GCE node
I20260808 18:36:09.636636 9592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:09.637291 9592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:09.638487 9592 hybrid_clock.cc:648] HybridClock initialized: now 1786214169638459 us; error 49 us; skew 500 ppm
Aug 08 18:36:09 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:09.642493 9592 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:09.643687 9592 webserver.cc:460] Webserver started at http://127.27.251.193:39463/ using document root <none> and password file <none>
I20260808 18:36:09.644415 9592 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:09.644550 9592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:09.644857 9592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:36:09 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:09.670729 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "288d9b46449645b6b97ec922915f42e7"
format_stamp: "Formatted at 2026-08-08 18:36:09 on dist-test-slave-fd7l"
server_key: "b9336ac3121dd680d2a409326e9a8ad2"
server_key_iv: "08837f5ab7411268c50a8d46baa7eaeb"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:09.671684 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "288d9b46449645b6b97ec922915f42e7"
format_stamp: "Formatted at 2026-08-08 18:36:09 on dist-test-slave-fd7l"
server_key: "b9336ac3121dd680d2a409326e9a8ad2"
server_key_iv: "08837f5ab7411268c50a8d46baa7eaeb"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:09.677731 9592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 18:36:09.708969 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:09.710872 9592 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260808 18:36:09.711032 9592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "288d9b46449645b6b97ec922915f42e7"
format_stamp: "Formatted at 2026-08-08 18:36:09 on dist-test-slave-fd7l"
server_key: "b9336ac3121dd680d2a409326e9a8ad2"
server_key_iv: "08837f5ab7411268c50a8d46baa7eaeb"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:09.711182 9592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 18:36:09 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:09.734709 9592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:09.738721 9592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:09.739033 9592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:09.739789 9592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:09.740936 9592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:09.741307 9592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:09.741454 9592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:09.741549 9592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:09.753439 9592 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:38215
I20260808 18:36:09.757870 9592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:09.767299 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 9592
I20260808 18:36:09.767445 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:36:09.767812 28655 external_mini_cluster.cc:1428] Setting key 931940e93837fcaaf88e231844b0a0f8
I20260808 18:36:09.759676 9731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:38215 every 8 connection(s)
Aug 08 18:36:09 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:09.788961 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:09.763407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:34877 (local address 127.27.251.254:45563)
0808 18:36:09.765856 (+ 2449us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:09.765861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:09.765878 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:36:09.765925 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:09.765928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:09.765980 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:09.766087 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:09.773102 (+ 7015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:09.773912 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:09.775703 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:09.776062 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:09.776149 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:09.783838 (+ 7689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:09.783843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:09.783846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:09.783882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:09.786051 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:09.786808 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:09.786812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:09.786814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:09.786916 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:09.787399 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:09.787405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:09.787408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:09.787665 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:09.787759 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:09.788593 (+ 834us) server_negotiation.cc:299] Negotiation successful
0808 18:36:09.788748 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2358,"thread_start_us":97,"threads_started":1}
I20260808 18:36:09.791100 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:09.762105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.193:34877)
0808 18:36:09.764191 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:09.764253 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:09.765394 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:09.766425 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:09.766434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:09.766824 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:09.772886 (+ 6062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:09.772913 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:09.774114 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:09.774118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:09.775554 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:09.775566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:09.776390 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:09.776395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:09.776649 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:09.777401 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:09.777478 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:09.779719 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:09.786234 (+ 6515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:09.786242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:09.786264 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:09.786675 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:09.787055 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:09.787058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:09.787060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:09.787273 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:09.787835 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:09.787842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:09.788241 (+ 399us) client_negotiation.cc:769] Sending connection context
0808 18:36:09.789503 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0808 18:36:09.789905 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"thread_start_us":178,"threads_started":1}
I20260808 18:36:09.793031 9732 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:09.793293 9732 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:09.793922 9732 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:09.795852 9509 ts_manager.cc:194] Registered new tserver with Master: 288d9b46449645b6b97ec922915f42e7 (127.27.251.193:38215)
I20260808 18:36:09.797462 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:34877
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:36:09.828680 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:09.998383 9739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:09.998741 9739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:09.998831 9739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:10.003428 9739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:10.003552 9739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:10.003656 9739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:10.010085 9739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:10.011379 9739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:10.012607 9739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:10.026396 9745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:10.027060 9748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:10.027837 9739 server_base.cc:1034] running on GCE node
W20260808 18:36:10.027776 9746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:10.028638 9739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:10.029507 9739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:10.030718 9739 hybrid_clock.cc:648] HybridClock initialized: now 1786214170030683 us; error 36 us; skew 500 ppm
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:10.034744 9739 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:10.035923 9739 webserver.cc:460] Webserver started at http://127.27.251.194:34047/ using document root <none> and password file <none>
I20260808 18:36:10.036746 9739 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:10.036928 9739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:10.037245 9739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:10.056042 9739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "fe9450e535e34a1ea7d3b0ac811f513b"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "2cb1b98d582bb13652fe7183be8c20f8"
server_key_iv: "81044b1ca2b147e0f7d5f14b1ef8e8c5"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.056878 9739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "fe9450e535e34a1ea7d3b0ac811f513b"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "2cb1b98d582bb13652fe7183be8c20f8"
server_key_iv: "81044b1ca2b147e0f7d5f14b1ef8e8c5"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.062398 9739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 18:36:10.091806 9756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.093751 9739 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260808 18:36:10.093916 9739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "fe9450e535e34a1ea7d3b0ac811f513b"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "2cb1b98d582bb13652fe7183be8c20f8"
server_key_iv: "81044b1ca2b147e0f7d5f14b1ef8e8c5"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.094079 9739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:10.120595 9739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:10.124884 9739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:10.125216 9739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:10.126021 9739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:10.127255 9739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:10.127372 9739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.127490 9739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:10.127568 9739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.140820 9739 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:44559
I20260808 18:36:10.142254 9739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:36:10.144501 9874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:44559 every 8 connection(s)
I20260808 18:36:10.151140 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 9739
I20260808 18:36:10.151292 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:36:10.151670 28655 external_mini_cluster.cc:1428] Setting key 069b93a772019b1c78d45ba994a60ad2
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:10.165782 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:10.150450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:35999 (local address 127.27.251.254:45563)
0808 18:36:10.150602 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:10.150608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:10.151448 (+ 840us) server_negotiation.cc:407] Connection header received
0808 18:36:10.152959 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:10.152963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:10.153031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:10.153089 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:10.154860 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.155716 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.157439 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.157803 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.157886 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:10.161147 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:10.161152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:10.161154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:10.161184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:10.163113 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:10.163842 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:10.163845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:10.163847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:10.163940 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:10.164396 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:10.164399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:10.164401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:10.164647 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:10.164743 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:10.165439 (+ 696us) server_negotiation.cc:299] Negotiation successful
0808 18:36:10.165586 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:36:10.167836 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:10.149091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.194:35999)
0808 18:36:10.151266 (+ 2175us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:10.151325 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:10.152702 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:10.153540 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:10.153551 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:10.153917 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:10.154679 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:10.154700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.155886 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.155889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:10.157292 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:10.157303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.158115 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.158119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:10.158318 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:10.159083 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:10.159104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:10.160932 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:10.163298 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:10.163304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:10.163322 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:10.163715 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:10.164074 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:10.164077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:10.164079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:10.164277 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:10.164814 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:10.164821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:10.165151 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 18:36:10.166367 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0808 18:36:10.166731 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"thread_start_us":164,"threads_started":1}
I20260808 18:36:10.171960 9875 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:10.172243 9875 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:10.172886 9875 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:10.174168 9509 ts_manager.cc:194] Registered new tserver with Master: fe9450e535e34a1ea7d3b0ac811f513b (127.27.251.194:44559)
I20260808 18:36:10.174942 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:35999
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:36:10.214866 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
W20260808 18:36:10.470590 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:10.471053 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:10.471167 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:10.476850 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:10.477053 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:10.477190 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:10.484898 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:10.486511 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:10.487887 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:10.500860 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:10.501376 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:10.501641 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:10.502735 9880 server_base.cc:1034] running on GCE node
I20260808 18:36:10.503290 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:10.504009 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:10.505187 9880 hybrid_clock.cc:648] HybridClock initialized: now 1786214170505107 us; error 93 us; skew 500 ppm
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:10.508818 9880 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:10.509822 9880 webserver.cc:460] Webserver started at http://127.27.251.195:40841/ using document root <none> and password file <none>
I20260808 18:36:10.510475 9880 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:10.510555 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:10.510771 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:10.528033 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "1caa89169fab5b953ae1600294aa2e7d"
server_key_iv: "aa96379dd93664b80d0652928ba31b1e"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.536839 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "1caa89169fab5b953ae1600294aa2e7d"
server_key_iv: "aa96379dd93664b80d0652928ba31b1e"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.544044 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260808 18:36:10.569964 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.571913 9880 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260808 18:36:10.572146 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "1caa89169fab5b953ae1600294aa2e7d"
server_key_iv: "aa96379dd93664b80d0652928ba31b1e"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.572403 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:10.592793 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:10.596676 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:10.596896 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:10.597666 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:10.598810 9880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:10.598893 9880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.599025 9880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:10.599073 9880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.615797 9880 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:33043
I20260808 18:36:10.616398 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:33043 every 8 connection(s)
I20260808 18:36:10.617342 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:10.626135 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 9880
I20260808 18:36:10.626286 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:36:10.626634 28655 external_mini_cluster.cc:1428] Setting key 3680a33cb58171bf10cb4a28be800457
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:10.643955 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:10.621788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:53399 (local address 127.27.251.254:45563)
0808 18:36:10.621924 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:10.621930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:10.622713 (+ 783us) server_negotiation.cc:407] Connection header received
0808 18:36:10.623781 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:10.623785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:10.623845 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:10.623935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:10.626929 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.627776 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.633459 (+ 5683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.633713 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.633841 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:10.637119 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:10.637124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:10.637127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:10.637158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:10.639205 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:10.642039 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:10.642043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:10.642046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:10.642143 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:10.642660 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:10.642663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:10.642666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:10.642915 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:10.643039 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:10.643639 (+ 600us) server_negotiation.cc:299] Negotiation successful
0808 18:36:10.643771 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:36:10.645121 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:10.620566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.195:53399)
0808 18:36:10.622532 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:10.622592 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:10.623561 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:10.624278 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:10.624288 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:10.625793 (+ 1505us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:10.626727 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:10.626751 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.628137 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.628140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:10.633315 (+ 5175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:10.633327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:10.633888 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:10.633892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:10.634130 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:10.634830 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:10.634853 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:10.636869 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:10.640976 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:10.640983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:10.641001 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:10.641898 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:10.642311 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:10.642315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:10.642317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:10.642522 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:10.643109 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:10.643113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:10.643367 (+ 254us) client_negotiation.cc:769] Sending connection context
0808 18:36:10.643570 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 18:36:10.643915 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":160,"threads_started":1}
I20260808 18:36:10.646476 10013 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:10.646746 10013 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:10.647367 10013 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:10.648562 9509 ts_manager.cc:194] Registered new tserver with Master: 9f238f6bdbfe46b7b4b8cd08e8c8a043 (127.27.251.195:33043)
I20260808 18:36:10.649420 9509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:53399
I20260808 18:36:10.652580 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:36:10.653378 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 9592
I20260808 18:36:10.664677 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 9739
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:10.676905 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 9880
I20260808 18:36:10.690470 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 9476
I20260808 18:36:10.700826 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45563
--webserver_interface=127.27.251.254
--webserver_port=34259
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45563
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:10.846186 10017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:10.846441 10017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:10.846477 10017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:10.850472 10017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:10.850522 10017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:10.850550 10017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:10.850566 10017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:10.850580 10017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:10.855985 10017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45563
--ranger_config_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:10.857056 10017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:10.858104 10017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:10.866869 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:10.866957 10017 server_base.cc:1034] running on GCE node
W20260808 18:36:10.867030 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:10.866875 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:10.867683 10017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:10.868835 10017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:10.870000 10017 hybrid_clock.cc:648] HybridClock initialized: now 1786214170869964 us; error 48 us; skew 500 ppm
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:10.872816 10017 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:10.873678 10017 webserver.cc:460] Webserver started at http://127.27.251.254:34259/ using document root <none> and password file <none>
I20260808 18:36:10.874219 10017 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:10.874310 10017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:36:10 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:10.928755 10017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:10.931180 10037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:10.932606 10017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:10.932754 10017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a0e158690fe04c879e893be751147b78"
format_stamp: "Formatted at 2026-08-08 18:36:08 on dist-test-slave-fd7l"
server_key: "8f4b3b06ef231f1a69228ed0f82195a8"
server_key_iv: "64629c1cd0bea6e9300a5c38bed9d6d1"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:10.933350 10017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:10.948845 10017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:10.951886 10017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:10.952101 10017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:10.959645 10017 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45563
I20260808 18:36:10.959673 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45563 every 8 connection(s)
I20260808 18:36:10.960834 10017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:10.965498 10091 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:36:10.967273 10091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Bootstrap starting.
I20260808 18:36:10.969772 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10017
I20260808 18:36:10.970361 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:38215
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:45563
--webserver_port=39463
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:10.986578 10091 log.cc:824] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:10.994179 10091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:36:10.994621 10091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Bootstrap complete.
I20260808 18:36:10.999255 10091 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:10.999688 10091 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e158690fe04c879e893be751147b78, State: Initialized, Role: FOLLOWER
I20260808 18:36:11.000278 10091 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:11.000404 10091 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:11.000499 10091 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:11.000631 10091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:36:11.002717 10091 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:11.003070 10091 leader_election.cc:302] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e158690fe04c879e893be751147b78; no voters:
I20260808 18:36:11.003415 10091 leader_election.cc:288] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:36:11.003562 10095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:36:11.003824 10095 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [term 2 LEADER]: Becoming Leader. State: Replica: a0e158690fe04c879e893be751147b78, State: Running, Role: LEADER
I20260808 18:36:11.004194 10095 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } }
I20260808 18:36:11.004791 10091 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:11.005637 10096 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a0e158690fe04c879e893be751147b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } } }
I20260808 18:36:11.005772 10096 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:11.006688 10104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:11.007088 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e158690fe04c879e893be751147b78. Latest consensus state: current_term: 2 leader_uuid: "a0e158690fe04c879e893be751147b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e158690fe04c879e893be751147b78" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45563 } } }
I20260808 18:36:11.007195 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:11.009974 10104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:11.010517 10091 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 18:36:11.010677 10104 catalog_manager.cc:1259] Loaded cluster ID: 862f717c42fd4b6294becb7101b7f6c7
I20260808 18:36:11.010725 10104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:11.013108 10104 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:11.013784 10104 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Loaded TSK: 0
I20260808 18:36:11.014931 10104 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:36:11.107752 10093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:11.108052 10093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:11.108158 10093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:11.113623 10093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:11.113750 10093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:11.113864 10093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:11.120158 10093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:38215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=39463
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:11.121392 10093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:11.122552 10093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:11.135900 10130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:11.135900 10133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:11.136075 10131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:11.137092 10093 server_base.cc:1034] running on GCE node
I20260808 18:36:11.137593 10093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:11.138373 10093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:11.139588 10093 hybrid_clock.cc:648] HybridClock initialized: now 1786214171139553 us; error 49 us; skew 500 ppm
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:11.142628 10093 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:11.143783 10093 webserver.cc:460] Webserver started at http://127.27.251.193:39463/ using document root <none> and password file <none>
I20260808 18:36:11.144421 10093 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:11.144564 10093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:11.180461 10093 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s
I20260808 18:36:11.186335 10141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.188046 10093 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260808 18:36:11.188275 10093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "288d9b46449645b6b97ec922915f42e7"
format_stamp: "Formatted at 2026-08-08 18:36:09 on dist-test-slave-fd7l"
server_key: "b9336ac3121dd680d2a409326e9a8ad2"
server_key_iv: "08837f5ab7411268c50a8d46baa7eaeb"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:11.189217 10093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:11.224303 10093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:11.228214 10093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:11.228448 10093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:11.229336 10093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:11.230571 10093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:11.230690 10093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.230793 10093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:11.230858 10093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.243287 10093 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:38215
I20260808 18:36:11.245029 10093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:11.247419 10257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:38215 every 8 connection(s)
I20260808 18:36:11.252856 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10093
I20260808 18:36:11.253408 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:44559
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:45563
--webserver_port=34047
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:11.276412 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.249962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.193:33227)
0808 18:36:11.250481 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:11.250536 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:11.251507 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:11.253328 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:11.253341 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:11.253714 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:11.254439 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.254460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.256964 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.256968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.258434 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.258448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.267182 (+ 8734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.267191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.267481 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.268258 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:11.268281 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:11.270159 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:11.272785 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.272791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.272807 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.273152 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.273557 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.273561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.273563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.273723 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.274451 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:11.274459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:11.274787 (+ 328us) client_negotiation.cc:769] Sending connection context
0808 18:36:11.275024 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 18:36:11.275378 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
I20260808 18:36:11.278805 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.251509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:33227 (local address 127.27.251.254:45563)
0808 18:36:11.252078 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:11.252096 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:11.252131 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:36:11.252354 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:11.252364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:11.252824 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:11.254749 (+ 1925us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:11.255131 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.256779 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.265924 (+ 9145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.266916 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.267833 (+ 917us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.270402 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:11.270409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:11.270428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:11.270462 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:11.272650 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.273295 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.273302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.273307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.273450 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.273831 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.273834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.273836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.274316 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:11.277442 (+ 3126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:11.277595 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 18:36:11.277891 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":143,"threads_started":1}
I20260808 18:36:11.281880 10258 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:11.282133 10258 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.282729 10258 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.284945 10258 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:45563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 18:36:11.285059 10258 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.285219 10258 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:11.286139 10258 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.286293 10258 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.287035 10258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:36:11.399989 10263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:11.400287 10263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:11.400367 10263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:11.405439 10263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:11.405539 10263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:11.405625 10263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:11.413097 10263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:44559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=34047
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:11.414702 10263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:11.415911 10263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:11.427680 10268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:11.428217 10263 server_base.cc:1034] running on GCE node
W20260808 18:36:11.427675 10269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:11.428282 10271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:11.428829 10263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:11.429694 10263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:11.430944 10263 hybrid_clock.cc:648] HybridClock initialized: now 1786214171430911 us; error 54 us; skew 500 ppm
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:11.434190 10263 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:11.435199 10263 webserver.cc:460] Webserver started at http://127.27.251.194:34047/ using document root <none> and password file <none>
I20260808 18:36:11.435781 10263 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:11.435887 10263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:11.454917 10263 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:11.458161 10279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.459714 10263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:11.459841 10263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "fe9450e535e34a1ea7d3b0ac811f513b"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "2cb1b98d582bb13652fe7183be8c20f8"
server_key_iv: "81044b1ca2b147e0f7d5f14b1ef8e8c5"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:11.460531 10263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:11.476382 10263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:11.480206 10263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:11.480472 10263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:11.481244 10263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:11.482384 10263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:11.482484 10263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.482581 10263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:11.482656 10263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.495507 10263 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:44559
I20260808 18:36:11.495600 10399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:44559 every 8 connection(s)
I20260808 18:36:11.496929 10263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:36:11.504679 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10263
I20260808 18:36:11.505267 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.251.212:51031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:33043
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:45563
--webserver_port=40841
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:11.535915 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.502975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:56375 (local address 127.27.251.254:45563)
0808 18:36:11.503163 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:11.503168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:11.521383 (+ 18215us) server_negotiation.cc:407] Connection header received
0808 18:36:11.522974 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:11.522980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:11.523059 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:11.523124 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:11.525402 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.526292 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.527915 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.528180 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.528297 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.531377 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:11.531384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:11.531388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:11.531424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:11.533387 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.534008 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.534012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.534014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.534097 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.534498 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.534501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.534503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.534736 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:11.534817 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:11.535580 (+ 763us) server_negotiation.cc:299] Negotiation successful
0808 18:36:11.535731 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:36:11.537909 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.504031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.194:56375)
0808 18:36:11.521088 (+ 17057us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:11.521196 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:11.522682 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:11.523699 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:11.523713 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:11.524122 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:11.525188 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.525217 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.526485 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.526490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.527782 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.527794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.528346 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.528350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.528626 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.529442 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:11.529470 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:11.531155 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:11.533520 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.533526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.533544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.533893 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.534218 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.534220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.534222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.534394 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.534884 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:11.534890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:11.535297 (+ 407us) client_negotiation.cc:769] Sending connection context
0808 18:36:11.536399 (+ 1102us) client_negotiation.cc:240] Negotiation successful
0808 18:36:11.536787 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16628,"thread_start_us":158,"threads_started":1}
I20260808 18:36:11.539450 10400 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:11.539702 10400 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.540253 10400 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.541996 10400 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:45563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 18:36:11.542109 10400 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.542290 10400 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:11.543071 10400 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.543220 10400 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.543967 10400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:36:11.665912 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:11.666296 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:11.666378 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:11.672247 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:11.672348 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:11.672464 10403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:11.678535 10403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.251.212:51031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:33043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=40841
--tserver_master_addrs=127.27.251.254:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:11.679733 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:11.680884 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:11.692781 10410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:11.693450 10412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:11.693817 10403 server_base.cc:1034] running on GCE node
W20260808 18:36:11.693962 10409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:11.695314 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:11.696141 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:11.697328 10403 hybrid_clock.cc:648] HybridClock initialized: now 1786214171697294 us; error 53 us; skew 500 ppm
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:11.700286 10403 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:11.701398 10403 webserver.cc:460] Webserver started at http://127.27.251.195:40841/ using document root <none> and password file <none>
I20260808 18:36:11.701975 10403 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:11.702049 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:11.721706 10403 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260808 18:36:11.724674 10420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.726344 10403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:36:11.726527 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043"
format_stamp: "Formatted at 2026-08-08 18:36:10 on dist-test-slave-fd7l"
server_key: "1caa89169fab5b953ae1600294aa2e7d"
server_key_iv: "aa96379dd93664b80d0652928ba31b1e"
server_key_version: "kuduclusterkey@0"
I20260808 18:36:11.727375 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:11.762430 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:11.766153 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:11.766402 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:11.767169 10403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:11.768234 10403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:11.768302 10403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.768358 10403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:11.768392 10403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:11.781605 10403 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:33043
I20260808 18:36:11.781630 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:33043 every 8 connection(s)
I20260808 18:36:11.782956 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:11.790477 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10403
Aug 08 18:36:11 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:11.800957 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.785766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45563 (local address 127.27.251.195:35407)
0808 18:36:11.786348 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:11.786407 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:11.787545 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:11.788377 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:11.788386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:11.788771 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:11.789440 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.789459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.790578 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.790583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.792078 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:11.792088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.792937 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.792945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:11.793195 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.793930 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:11.793955 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:11.795457 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:11.797588 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.797595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.797612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.797974 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.798297 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:11.798300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:11.798302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:11.798450 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:11.798978 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:11.798983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:11.799332 (+ 349us) client_negotiation.cc:769] Sending connection context
0808 18:36:11.799563 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 18:36:11.799946 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":140,"threads_started":1}
I20260808 18:36:11.801676 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.785207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:35407 (local address 127.27.251.254:45563)
0808 18:36:11.785354 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:11.785359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:11.786528 (+ 1169us) server_negotiation.cc:407] Connection header received
0808 18:36:11.787813 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:11.787817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:11.787875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:11.787938 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:11.789611 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.790401 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.792221 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.792571 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.792681 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.795626 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:11.795630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:11.795634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:11.795665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:11.797450 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.798092 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.798096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.798100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.798195 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.798550 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.798555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.798557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.798824 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:11.798929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:11.801424 (+ 2495us) server_negotiation.cc:299] Negotiation successful
0808 18:36:11.801560 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:36:11.802632 10542 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45563
I20260808 18:36:11.802872 10542 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.803390 10542 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.805298 10542 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:45563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 18:36:11.805437 10542 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.805653 10542 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:11.806634 10542 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:11.806849 10542 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
W20260808 18:36:11.807618 10542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:36:11.808876 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:11.791206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.27.251.254:45563)
0808 18:36:11.792346 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:11.792355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:11.792657 (+ 302us) server_negotiation.cc:407] Connection header received
0808 18:36:11.792769 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:11.792774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:11.792830 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:11.792930 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:11.794704 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.795474 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.798118 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:11.798474 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:11.798593 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:11.800180 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:11.800184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:11.800187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:11.800214 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:11.805519 (+ 5305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.806070 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.806073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.806075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.806226 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:11.807931 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:11.807935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:11.807937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:11.808176 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:11.808405 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:11.808554 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 18:36:11.808692 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"spinlock_wait_cycles":60160,"thread_start_us":101,"threads_started":1}
Aug 08 18:36:12 dist-test-slave-fd7l krb5kdc[9012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786214172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:12.287253 10258 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:12.287577 10258 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:12.288815 10054 ts_manager.cc:194] Registered new tserver with Master: 288d9b46449645b6b97ec922915f42e7 (127.27.251.193:38215)
I20260808 18:36:12.290052 10054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:33227
I20260808 18:36:12.544189 10400 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:12.544548 10400 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:12.545300 10054 ts_manager.cc:194] Registered new tserver with Master: fe9450e535e34a1ea7d3b0ac811f513b (127.27.251.194:44559)
I20260808 18:36:12.545953 10054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:56375
Aug 08 18:36:12 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:12.807873 10542 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:12.808197 10542 heartbeater.cc:511] Master 127.27.251.254:45563 requested a full tablet report, sending...
I20260808 18:36:12.809075 10054 ts_manager.cc:194] Registered new tserver with Master: 9f238f6bdbfe46b7b4b8cd08e8c8a043 (127.27.251.195:33043)
I20260808 18:36:12.809899 10054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:35407
I20260808 18:36:12.810320 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:36:12.822175 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.812064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.27.251.254:45563)
0808 18:36:12.812658 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.812663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.812833 (+ 170us) server_negotiation.cc:407] Connection header received
0808 18:36:12.812951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.812954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.813001 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.813115 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:12.814082 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.814893 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.816100 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.816463 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.816657 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.818474 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:12.818477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:12.818480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:12.818507 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:12.820283 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:12.820795 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:12.820799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:12.820801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:12.820890 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:12.821169 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:12.821172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:12.821174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:12.821367 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:12.821479 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.821822 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.821963 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":80,"threads_started":1}
I20260808 18:36:12.826432 10054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:36:12.831364 10054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:36:12.851246 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.843680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:33043 (local address 127.0.0.1:37734)
0808 18:36:12.844109 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:12.844161 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:12.844288 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:12.845266 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:12.845273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:12.845297 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:12.845446 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.845454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.847362 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.847366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.848726 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.848738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.850689 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.850693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.850811 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.850860 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 18:36:12.850952 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:36:12.851089 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":121,"threads_started":1}
I20260808 18:36:12.852337 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.843800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.27.251.195:33043)
0808 18:36:12.844743 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.844751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.844776 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:36:12.844856 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.844861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.845084 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.845616 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:12.845758 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.847236 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.848892 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.850546 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.851881 (+ 1335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.852046 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.852101 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.852194 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":81,"threads_started":1}
I20260808 18:36:12.853482 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.845994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:38215 (local address 127.0.0.1:60718)
0808 18:36:12.847756 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:12.847767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:12.847845 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:12.849439 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:12.849442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:12.849454 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:12.849513 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.849517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.850825 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.850828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.851974 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.851985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.853078 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.853082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.853178 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.853204 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:36:12.853281 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:36:12.853345 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"thread_start_us":102,"threads_started":1}
I20260808 18:36:12.854914 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.846394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.27.251.193:38215)
0808 18:36:12.847639 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.847645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.847927 (+ 282us) server_negotiation.cc:407] Connection header received
0808 18:36:12.848002 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.848008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.848268 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.848407 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:12.849623 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.850691 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.852123 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.852939 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.853064 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.853182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.854682 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.854778 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":104,"threads_started":1}
I20260808 18:36:12.855659 10453 tablet_service.cc:1467] Processing CreateTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed (DEFAULT_TABLE table=test-table [id=758be698b28149a18b1883eed2cac2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:12.856863 10453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fabed63a4d340dcbd30f76285a4a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:12.857067 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.844883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:44559 (local address 127.0.0.1:44960)
0808 18:36:12.847320 (+ 2437us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:12.847333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:12.847414 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:12.849238 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:12.849242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:12.849258 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:12.849340 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.849345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.854528 (+ 5183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.854531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.855608 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.855620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.856656 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.856660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.856751 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.856775 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 18:36:12.856858 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 18:36:12.856923 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2384,"thread_start_us":91,"threads_started":1}
I20260808 18:36:12.857586 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.845291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.27.251.194:44559)
0808 18:36:12.848704 (+ 3413us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.848711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.848733 (+ 22us) server_negotiation.cc:407] Connection header received
0808 18:36:12.848802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.848806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.848969 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.849102 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:12.853257 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.854392 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.855760 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.856540 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.856661 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.856775 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.857347 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.857444 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3310,"thread_start_us":91,"threads_started":1}
I20260808 18:36:12.858867 10170 tablet_service.cc:1467] Processing CreateTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed (DEFAULT_TABLE table=test-table [id=758be698b28149a18b1883eed2cac2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:12.859932 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fabed63a4d340dcbd30f76285a4a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:12.863464 10313 tablet_service.cc:1467] Processing CreateTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed (DEFAULT_TABLE table=test-table [id=758be698b28149a18b1883eed2cac2c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:12.864573 10313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fabed63a4d340dcbd30f76285a4a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:12.866767 10575 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Bootstrap starting.
I20260808 18:36:12.869181 10575 tablet_bootstrap.cc:654] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:12.870131 10575 log.cc:824] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:12.870895 10576 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Bootstrap starting.
I20260808 18:36:12.873158 10576 tablet_bootstrap.cc:654] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:12.875430 10575 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: No bootstrap required, opened a new log
I20260808 18:36:12.875706 10575 ts_tablet_manager.cc:1397] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 18:36:12.879586 10575 raft_consensus.cc:348] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.880023 10575 raft_consensus.cc:374] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:12.880126 10575 raft_consensus.cc:729] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f238f6bdbfe46b7b4b8cd08e8c8a043, State: Initialized, Role: FOLLOWER
I20260808 18:36:12.880769 10575 consensus_queue.cc:260] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.881966 10576 log.cc:824] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:12.883267 10575 ts_tablet_manager.cc:1428] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:36:12.884104 10542 heartbeater.cc:503] Master 127.27.251.254:45563 was elected leader, sending a full tablet report...
I20260808 18:36:12.888031 10578 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Bootstrap starting.
I20260808 18:36:12.890113 10576 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: No bootstrap required, opened a new log
I20260808 18:36:12.890345 10576 ts_tablet_manager.cc:1397] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260808 18:36:12.890708 10578 tablet_bootstrap.cc:654] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:12.891578 10578 log.cc:824] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:12.895124 10576 raft_consensus.cc:348] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.895587 10576 raft_consensus.cc:374] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:12.895665 10576 raft_consensus.cc:729] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288d9b46449645b6b97ec922915f42e7, State: Initialized, Role: FOLLOWER
I20260808 18:36:12.896214 10576 consensus_queue.cc:260] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.897375 10576 ts_tablet_manager.cc:1428] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:36:12.897797 10258 heartbeater.cc:503] Master 127.27.251.254:45563 was elected leader, sending a full tablet report...
I20260808 18:36:12.898907 10578 tablet_bootstrap.cc:492] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: No bootstrap required, opened a new log
I20260808 18:36:12.899247 10578 ts_tablet_manager.cc:1397] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 18:36:12.903461 10578 raft_consensus.cc:348] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.903924 10578 raft_consensus.cc:374] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:12.905041 10578 raft_consensus.cc:729] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9450e535e34a1ea7d3b0ac811f513b, State: Initialized, Role: FOLLOWER
I20260808 18:36:12.905638 10578 consensus_queue.cc:260] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.906517 10400 heartbeater.cc:503] Master 127.27.251.254:45563 was elected leader, sending a full tablet report...
I20260808 18:36:12.906854 10578 ts_tablet_manager.cc:1428] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
Aug 08 18:36:12 dist-test-slave-fd7l krb5kdc[9012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for HTTP/127.27.251.212@KRBTEST.COM
I20260808 18:36:12.927366 10583 raft_consensus.cc:482] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:12.927621 10583 raft_consensus.cc:504] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.929281 10583 leader_election.cc:288] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f238f6bdbfe46b7b4b8cd08e8c8a043 (127.27.251.195:33043), fe9450e535e34a1ea7d3b0ac811f513b (127.27.251.194:44559)
I20260808 18:36:12.936812 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.929868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:59167 (local address 127.27.251.195:33043)
0808 18:36:12.929990 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.929995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.931762 (+ 1767us) server_negotiation.cc:407] Connection header received
0808 18:36:12.931925 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.931929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.931979 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.932028 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:12.932455 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.933263 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.935451 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.936025 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.936119 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.936170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.936585 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.936667 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:36:12.937211 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.930334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:41333 (local address 127.27.251.194:44559)
0808 18:36:12.930498 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.930504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.932099 (+ 1595us) server_negotiation.cc:407] Connection header received
0808 18:36:12.932330 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.932334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.932397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.932452 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:12.932986 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.933755 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.935707 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.936350 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.936455 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.936567 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.936989 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.937082 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 18:36:12.937325 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.929451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:33043 (local address 127.27.251.193:59167)
0808 18:36:12.931632 (+ 2181us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:12.931646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:12.931775 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:12.932178 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:12.932181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:12.932199 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:12.932341 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.932346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.933548 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.933551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.935305 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.935317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.936264 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.936267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.936361 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.936382 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:36:12.937085 (+ 703us) client_negotiation.cc:240] Negotiation successful
0808 18:36:12.937162 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2107,"mutex_wait_us":177,"thread_start_us":128,"threads_started":1}
I20260808 18:36:12.938017 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fabed63a4d340dcbd30f76285a4a6ed" candidate_uuid: "288d9b46449645b6b97ec922915f42e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" is_pre_election: true
I20260808 18:36:12.938103 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.929798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:44559 (local address 127.27.251.193:41333)
0808 18:36:12.931974 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:12.931992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:12.932193 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:12.932670 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:12.932673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:12.932694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:12.932824 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.932832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.933914 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.933921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.935551 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:12.935565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.936686 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.936690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:12.936816 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.936843 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:36:12.937862 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0808 18:36:12.937948 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":115,"threads_started":1}
I20260808 18:36:12.938349 10488 raft_consensus.cc:2454] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288d9b46449645b6b97ec922915f42e7 in term 0.
I20260808 18:36:12.938755 10350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fabed63a4d340dcbd30f76285a4a6ed" candidate_uuid: "288d9b46449645b6b97ec922915f42e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" is_pre_election: true
I20260808 18:36:12.938858 10145 leader_election.cc:302] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288d9b46449645b6b97ec922915f42e7, 9f238f6bdbfe46b7b4b8cd08e8c8a043; no voters:
I20260808 18:36:12.939054 10350 raft_consensus.cc:2454] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288d9b46449645b6b97ec922915f42e7 in term 0.
I20260808 18:36:12.939383 10583 raft_consensus.cc:2781] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:36:12.939497 10583 raft_consensus.cc:482] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:36:12.939558 10583 raft_consensus.cc:3037] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:12.941079 10583 raft_consensus.cc:504] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.941510 10583 leader_election.cc:288] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [CANDIDATE]: Term 1 election: Requested vote from peers 9f238f6bdbfe46b7b4b8cd08e8c8a043 (127.27.251.195:33043), fe9450e535e34a1ea7d3b0ac811f513b (127.27.251.194:44559)
I20260808 18:36:12.942711 10350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fabed63a4d340dcbd30f76285a4a6ed" candidate_uuid: "288d9b46449645b6b97ec922915f42e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9450e535e34a1ea7d3b0ac811f513b"
I20260808 18:36:12.942860 10350 raft_consensus.cc:3037] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:12.944430 10350 raft_consensus.cc:2454] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288d9b46449645b6b97ec922915f42e7 in term 1.
I20260808 18:36:12.944655 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fabed63a4d340dcbd30f76285a4a6ed" candidate_uuid: "288d9b46449645b6b97ec922915f42e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043"
I20260808 18:36:12.944823 10488 raft_consensus.cc:3037] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:12.946455 10488 raft_consensus.cc:2454] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288d9b46449645b6b97ec922915f42e7 in term 1.
I20260808 18:36:12.946956 10145 leader_election.cc:302] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288d9b46449645b6b97ec922915f42e7, 9f238f6bdbfe46b7b4b8cd08e8c8a043; no voters:
I20260808 18:36:12.947249 10583 raft_consensus.cc:2781] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:12.947541 10583 raft_consensus.cc:686] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [term 1 LEADER]: Becoming Leader. State: Replica: 288d9b46449645b6b97ec922915f42e7, State: Running, Role: LEADER
I20260808 18:36:12.947956 10583 consensus_queue.cc:237] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } }
I20260808 18:36:12.952538 10054 catalog_manager.cc:5495] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 288d9b46449645b6b97ec922915f42e7 (127.27.251.193). New cstate: current_term: 1 leader_uuid: "288d9b46449645b6b97ec922915f42e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288d9b46449645b6b97ec922915f42e7" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 38215 } health_report { overall_health: HEALTHY } } }
I20260808 18:36:12.979197 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:12.973726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.27.251.193:38215)
0808 18:36:12.974384 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:12.974389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:12.974405 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:36:12.974498 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:12.974501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:12.974573 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:12.974703 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:36:12.975036 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.975828 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.977071 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:12.977391 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:12.977480 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:12.977888 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:36:12.978444 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:36:12.978571 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:12.978973 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 18:36:12.979036 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554}
I20260808 18:36:12.986559 10350 raft_consensus.cc:1261] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 1 FOLLOWER]: Refusing update from remote peer 288d9b46449645b6b97ec922915f42e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:12.987432 10587 consensus_queue.cc:1035] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe9450e535e34a1ea7d3b0ac811f513b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:12.995020 10488 raft_consensus.cc:1261] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 1 FOLLOWER]: Refusing update from remote peer 288d9b46449645b6b97ec922915f42e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:12.995952 10587 consensus_queue.cc:1035] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f238f6bdbfe46b7b4b8cd08e8c8a043" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 18:36:12.999956 10401 tablet.cc:2369] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:36:13.000679 10260 tablet.cc:2369] T 2fabed63a4d340dcbd30f76285a4a6ed P 288d9b46449645b6b97ec922915f42e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:13.008092 10590 mvcc.cc:205] Tried to move back new op lower bound from 7316333252544966656 to 7316333252401053696. Current Snapshot: MvccSnapshot[applied={T|T < 7316333252544966656}]
I20260808 18:36:13.016362 10591 mvcc.cc:205] Tried to move back new op lower bound from 7316333252544966656 to 7316333252401053696. Current Snapshot: MvccSnapshot[applied={T|T < 7316333252544966656}]
I20260808 18:36:13.016793 10593 mvcc.cc:205] Tried to move back new op lower bound from 7316333252544966656 to 7316333252401053696. Current Snapshot: MvccSnapshot[applied={T|T < 7316333252544966656}]
I20260808 18:36:13.030498 10054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:13.030701 10054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:13.033670 10054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0e158690fe04c879e893be751147b78: Sending DeleteTablet for 3 replicas of tablet 2fabed63a4d340dcbd30f76285a4a6ed
I20260808 18:36:13.034888 10170 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:13 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:60718
I20260808 18:36:13.035655 10453 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:13 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37734
I20260808 18:36:13.036746 10313 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fabed63a4d340dcbd30f76285a4a6ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:13 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:44960
I20260808 18:36:13.037986 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10093
I20260808 18:36:13.044073 10601 tablet_replica.cc:330] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: stopping tablet replica
I20260808 18:36:13.044435 10601 raft_consensus.cc:2229] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:13.044433 10600 tablet_replica.cc:330] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: stopping tablet replica
I20260808 18:36:13.044836 10601 raft_consensus.cc:2258] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:13.045253 10600 raft_consensus.cc:2229] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:13.049052 10601 ts_tablet_manager.cc:1905] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:13.049542 10600 raft_consensus.cc:2258] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:13.051054 10600 ts_tablet_manager.cc:1905] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:13.052086 10601 ts_tablet_manager.cc:1918] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:13.052193 10601 log.cc:1196] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/2fabed63a4d340dcbd30f76285a4a6ed
I20260808 18:36:13.054069 10600 ts_tablet_manager.cc:1918] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:13.056838 10601 ts_tablet_manager.cc:1939] T 2fabed63a4d340dcbd30f76285a4a6ed P fe9450e535e34a1ea7d3b0ac811f513b: Deleting consensus metadata
I20260808 18:36:13.057137 10600 log.cc:1196] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/2fabed63a4d340dcbd30f76285a4a6ed
I20260808 18:36:13.057750 10600 ts_tablet_manager.cc:1939] T 2fabed63a4d340dcbd30f76285a4a6ed P 9f238f6bdbfe46b7b4b8cd08e8c8a043: Deleting consensus metadata
I20260808 18:36:13.058316 10039 catalog_manager.cc:4841] TS fe9450e535e34a1ea7d3b0ac811f513b (127.27.251.194:44559): tablet 2fabed63a4d340dcbd30f76285a4a6ed (table test-table [id=758be698b28149a18b1883eed2cac2c3]) successfully deleted
I20260808 18:36:13.059190 10041 catalog_manager.cc:4841] TS 9f238f6bdbfe46b7b4b8cd08e8c8a043 (127.27.251.195:33043): tablet 2fabed63a4d340dcbd30f76285a4a6ed (table test-table [id=758be698b28149a18b1883eed2cac2c3]) successfully deleted
W20260808 18:36:13.060437 10038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:38215 (error 108)
I20260808 18:36:13.062201 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10263
I20260808 18:36:13.062218 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:13.061862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:38215 (local address 127.0.0.1:60744)
0808 18:36:13.062025 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:13.062107 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 18:36:13.064306 10038 catalog_manager.cc:4584] TS 288d9b46449645b6b97ec922915f42e7 (127.27.251.193:38215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fabed63a4d340dcbd30f76285a4a6ed: Network error: Client connection negotiation failed: client connection to 127.27.251.193:38215: connect: Connection refused (error 111)
I20260808 18:36:13.073776 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10403
I20260808 18:36:13.085367 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10017
I20260808 18:36:13.096382 28655 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 18:36:13.441378 28655 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 18:36:13.441501 28655 mini_ranger.cc:66] Stopping Ranger...
I20260808 18:36:13.786638 28655 mini_ranger.cc:68] Stopped Ranger
2026-08-08 18:36:13.786 UTC [9042] LOG: received smart shutdown request
2026-08-08 18:36:13.791 UTC [9042] LOG: background worker "logical replication launcher" (PID 9051) exited with exit code 1
2026-08-08 18:36:13.792 UTC [9046] LOG: shutting down
2026-08-08 18:36:13.977 UTC [9042] LOG: database system is shut down
2026-08-08T18:36:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51610 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:36:14 dist-test-slave-fd7l krb5kdc[10611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:36:14 dist-test-slave-fd7l krb5kdc[10611](info): set up 2 sockets
Aug 08 18:36:14 dist-test-slave-fd7l krb5kdc[10611](info): commencing operation
krb5kdc: starting...
W20260808 18:36:15.255867 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:36:15.384199 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37973
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:15.498600 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:15.498876 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:15.498955 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:15.503052 10625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:15.503125 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:15.503163 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:15.503216 10625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:15.503257 10625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:15.508399 10625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:15.509444 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:15.510448 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:15.519243 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.519251 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.519258 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:15.519312 10625 server_base.cc:1034] running on GCE node
I20260808 18:36:15.520185 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:15.521595 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:15.522833 10625 hybrid_clock.cc:648] HybridClock initialized: now 1786214175522801 us; error 42 us; skew 500 ppm
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:15.525765 10625 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:15.526650 10625 webserver.cc:460] Webserver started at http://127.27.251.254:38875/ using document root <none> and password file <none>
I20260808 18:36:15.527227 10625 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:15.527319 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:15.527551 10625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:15.529651 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "ee4b0de1ed914b3d9861bb45f320c7ae"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "c238c15b2d93ad8a5f4778c3ced8da93"
server_key_iv: "8d342fd00ca41218f267e15b3f498227"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.530241 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "ee4b0de1ed914b3d9861bb45f320c7ae"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "c238c15b2d93ad8a5f4778c3ced8da93"
server_key_iv: "8d342fd00ca41218f267e15b3f498227"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.534008 10625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:15.536695 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:15.538046 10625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:15.538188 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "ee4b0de1ed914b3d9861bb45f320c7ae"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "c238c15b2d93ad8a5f4778c3ced8da93"
server_key_iv: "8d342fd00ca41218f267e15b3f498227"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.538300 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:15.570024 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:15.573100 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:15.573320 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:15.580596 10625 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:37973
I20260808 18:36:15.580895 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:37973 every 8 connection(s)
I20260808 18:36:15.581595 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:15.584378 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:15.588043 10696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:15.591065 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10625
I20260808 18:36:15.591193 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:36:15.591490 28655 external_mini_cluster.cc:1428] Setting key e812eb7107b987a0756d52e9e4f2f0b9
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.253@KRBTEST.COM" created.
I20260808 18:36:15.604594 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.590203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:37973 (local address 127.0.0.1:60064)
0808 18:36:15.590678 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.590721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:15.591820 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:15.592588 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:15.592597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:15.593087 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:15.593473 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.593512 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.595246 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.595248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.595964 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.595972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.596994 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.596997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.597074 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.597715 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:15.597732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:15.599402 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:15.601732 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.601741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.601748 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.602004 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.602384 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.602390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.602395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.602530 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.603077 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:15.603082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:15.603203 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 18:36:15.603378 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 18:36:15.603650 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260808 18:36:15.604595 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.590915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.27.251.254:37973)
0808 18:36:15.591192 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:15.591206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:15.591259 (+ 53us) server_negotiation.cc:407] Connection header received
0808 18:36:15.592142 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:15.592146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:15.592320 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:15.592477 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:15.593699 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.595121 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.596088 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.596844 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.597033 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.599577 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:15.599585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:15.599599 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:15.599627 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:15.601633 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.602127 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.602132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.602133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.602249 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.602615 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.602618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.602619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.602969 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:15.603073 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:15.603475 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 18:36:15.603721 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
I20260808 18:36:15.606994 10696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:15.607752 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.607510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43602)
0808 18:36:15.607634 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.607701 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 18:36:15.607934 10642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
I20260808 18:36:15.608832 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.608553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43610)
0808 18:36:15.608686 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.608752 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 18:36:15.608983 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:46109: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
I20260808 18:36:15.622064 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:46109
--webserver_interface=127.27.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:15.658252 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } attempt: 1
I20260808 18:36:15.659198 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.658838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43620)
0808 18:36:15.659025 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.659102 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 18:36:15.659965 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.659728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43630)
0808 18:36:15.659837 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.659890 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:36:15.660104 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:46109: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
I20260808 18:36:15.728297 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } attempt: 2
I20260808 18:36:15.729206 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.728855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43644)
0808 18:36:15.729036 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.729111 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 18:36:15.730013 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.729763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43660)
0808 18:36:15.729884 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.729944 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 18:36:15.730166 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:46109: Network error: Client connection negotiation failed: client connection to 127.27.251.253:46109: connect: Connection refused (error 111)
W20260808 18:36:15.740061 10704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:15.740284 10704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:15.740358 10704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:15.744652 10704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:15.744699 10704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:15.744719 10704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:15.744771 10704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:15.744805 10704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:15.750000 10704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:15.751035 10704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:15.752038 10704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:15.760423 10709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.760488 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.760429 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:15.760670 10704 server_base.cc:1034] running on GCE node
I20260808 18:36:15.761264 10704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:15.762403 10704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:15.763566 10704 hybrid_clock.cc:648] HybridClock initialized: now 1786214175763550 us; error 27 us; skew 500 ppm
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:15.766603 10704 init.cc:377] Logged in from keytab as kudu/127.27.251.253@KRBTEST.COM (short username kudu)
I20260808 18:36:15.767452 10704 webserver.cc:460] Webserver started at http://127.27.251.253:44383/ using document root <none> and password file <none>
I20260808 18:36:15.767997 10704 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:15.768072 10704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:15.768281 10704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:15.770364 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data/instance:
uuid: "04696b8b0ac64507b0393710a116dee1"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "b92393b50dfbf9085433f527b1274ba7"
server_key_iv: "28a89f126506ac3cf6f0c7de4b456742"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.770970 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal/instance:
uuid: "04696b8b0ac64507b0393710a116dee1"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "b92393b50dfbf9085433f527b1274ba7"
server_key_iv: "28a89f126506ac3cf6f0c7de4b456742"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.774621 10704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:36:15.777068 10719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:15.778275 10704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:15.778422 10704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "04696b8b0ac64507b0393710a116dee1"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "b92393b50dfbf9085433f527b1274ba7"
server_key_iv: "28a89f126506ac3cf6f0c7de4b456742"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.778533 10704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:15.796729 10704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:15.799921 10704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:15.800128 10704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:15.807360 10704 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.253:46109
I20260808 18:36:15.807695 10772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.253:46109 every 8 connection(s)
I20260808 18:36:15.808343 10704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
I20260808 18:36:15.809253 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10704
I20260808 18:36:15.809391 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-1/wal/instance
I20260808 18:36:15.809710 28655 external_mini_cluster.cc:1428] Setting key 9309b99f27d1d3227e19df0d9b0d618d
I20260808 18:36:15.811971 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:15.815863 10773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.252@KRBTEST.COM" created.
I20260808 18:36:15.821328 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } attempt: 3
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:36:15.832249 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.817826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.27.251.254:37973)
0808 18:36:15.817992 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:15.817997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:15.818861 (+ 864us) server_negotiation.cc:407] Connection header received
0808 18:36:15.819764 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:15.819768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:15.819820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:15.819898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:15.821184 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.822065 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.823782 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.824149 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.824261 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.827395 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:15.827399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:15.827402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:15.827426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:15.829879 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830474 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.830478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.830479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.830579 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830919 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.830922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.830924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.831107 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:15.831216 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:15.831805 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 18:36:15.832082 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:36:15.832392 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.821772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43664)
0808 18:36:15.821961 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.821977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:15.822081 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:15.822749 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:15.822754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:15.822989 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:15.823175 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.823184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.824494 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.824497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.825300 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.825310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.825953 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.825957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.826063 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.826574 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:15.826590 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:15.828203 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:15.830338 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.830347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.830602 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.830948 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.830953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.831041 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.831677 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:15.831680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:15.831783 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 18:36:15.831904 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 18:36:15.832193 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":125,"spinlock_wait_cycles":1792}
I20260808 18:36:15.833221 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.818251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:37973 (local address 127.0.0.1:60068)
0808 18:36:15.818709 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.818751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:15.819568 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:15.820069 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:15.820075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:15.820393 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:15.821016 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.821047 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.822239 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.822243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.823646 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.823659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.824314 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.824322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.824596 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.825293 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:15.825318 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:15.827261 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:15.830019 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830029 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.830041 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.830370 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.830674 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.830680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.830820 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.831220 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:15.831229 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:15.831572 (+ 343us) client_negotiation.cc:769] Sending connection context
0808 18:36:15.831748 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 18:36:15.832171 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260808 18:36:15.833228 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.821831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.27.251.253:46109)
0808 18:36:15.822303 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:15.822315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:15.822335 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:36:15.822371 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:15.822376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:15.822569 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:15.822732 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:15.823308 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.824375 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.825424 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.825824 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.825950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.828371 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:15.828379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:15.828394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:15.828427 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:15.830221 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.830743 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.830749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.830751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.830839 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.831136 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.831141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.831142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.831565 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:15.831716 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:15.831990 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 18:36:15.832288 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":373,"thread_start_us":125,"threads_started":1}
I20260808 18:36:15.834439 10773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:15.835237 10696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:15.835917 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.835672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48680)
0808 18:36:15.835807 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.835865 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:36:15.836628 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.836353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48694)
0808 18:36:15.836495 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.836572 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260808 18:36:15.836829 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:36:15.839979 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:38891
--webserver_interface=127.27.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:15.843250 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.834969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43678)
0808 18:36:15.835059 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.835072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:15.835151 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:15.835599 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:15.835605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:15.835767 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:15.835876 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.835881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.836838 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.836841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.837452 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:15.837461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.837931 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.837935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:15.838032 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.838530 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:15.838544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:15.839894 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:15.841768 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.841771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.841772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.841957 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.842270 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:15.842272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:15.842273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:15.842348 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:15.842729 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:15.842732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:15.842820 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 18:36:15.842927 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:36:15.843065 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 18:36:15.843259 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.835005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.27.251.253:46109)
0808 18:36:15.835329 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:15.835333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:15.835346 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:36:15.835378 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:15.835380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:15.835420 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:15.835495 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:15.835995 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.836729 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.837567 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:15.837825 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:15.837931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:15.840026 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:15.840029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:15.840032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:15.840054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:15.841650 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.842054 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.842060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.842063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.842172 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:15.842446 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:15.842451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:15.842453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:15.842633 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:15.842739 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:15.842972 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 18:36:15.843124 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":249}
I20260808 18:36:15.844228 10773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:15.844978 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.844732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48710)
0808 18:36:15.844857 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.844925 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:36:15.845706 10720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
I20260808 18:36:15.846642 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.846398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48720)
0808 18:36:15.846512 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.846579 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 18:36:15.846807 10773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
I20260808 18:36:15.868016 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 1
I20260808 18:36:15.868836 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.868548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48734)
0808 18:36:15.868703 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.868762 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 18:36:15.869601 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.869377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48742)
0808 18:36:15.869491 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.869543 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:36:15.869740 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
I20260808 18:36:15.896063 10773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 1
I20260808 18:36:15.896942 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.896645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48748)
0808 18:36:15.896799 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.896862 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 18:36:15.897740 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.897491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48752)
0808 18:36:15.897638 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.897681 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:36:15.897903 10773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
I20260808 18:36:15.944895 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 2
I20260808 18:36:15.945762 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.945425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48764)
0808 18:36:15.945615 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.945685 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 18:36:15.946565 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.946335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48774)
0808 18:36:15.946453 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.946506 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:36:15.946687 10696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
W20260808 18:36:15.957598 10779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:15.957876 10779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:15.957971 10779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:15.962028 10779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:15.962101 10779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:15.962141 10779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:15.962208 10779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:15.962247 10779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:15.966058 10773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 2
I20260808 18:36:15.966849 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.966560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48784)
0808 18:36:15.966704 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.966773 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 18:36:15.967592 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:15.967374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48794)
0808 18:36:15.967484 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:15.967543 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:36:15.967747 10773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38891: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38891: connect: Connection refused (error 111)
I20260808 18:36:15.968025 10779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:37973,127.27.251.253:46109,127.27.251.252:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:15.969162 10779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:15.970193 10779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:15.979136 10788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.979172 10786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:15.979138 10785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:15.979501 10779 server_base.cc:1034] running on GCE node
I20260808 18:36:15.980042 10779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:15.981264 10779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:15.982461 10779 hybrid_clock.cc:648] HybridClock initialized: now 1786214175982429 us; error 44 us; skew 500 ppm
Aug 08 18:36:15 dist-test-slave-fd7l krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:15.985466 10779 init.cc:377] Logged in from keytab as kudu/127.27.251.252@KRBTEST.COM (short username kudu)
I20260808 18:36:15.986306 10779 webserver.cc:460] Webserver started at http://127.27.251.252:39019/ using document root <none> and password file <none>
I20260808 18:36:15.986824 10779 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:15.986882 10779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:15.987054 10779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:15.989135 10779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data/instance:
uuid: "c05b0c7bd8404c1099d6f70c81cf19bb"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "8a52e5f3fd003ec11ed4d6dd627aa789"
server_key_iv: "e4f5b17dd92becaad75fbc4e8fae4d1c"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.989653 10779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal/instance:
uuid: "c05b0c7bd8404c1099d6f70c81cf19bb"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "8a52e5f3fd003ec11ed4d6dd627aa789"
server_key_iv: "e4f5b17dd92becaad75fbc4e8fae4d1c"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.993314 10779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:15.995625 10795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:15.996887 10779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:15.997053 10779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "c05b0c7bd8404c1099d6f70c81cf19bb"
format_stamp: "Formatted at 2026-08-08 18:36:15 on dist-test-slave-fd7l"
server_key: "8a52e5f3fd003ec11ed4d6dd627aa789"
server_key_iv: "e4f5b17dd92becaad75fbc4e8fae4d1c"
server_key_version: "encryptionkey@0"
I20260808 18:36:15.997146 10779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:16.020401 10779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:16.023783 10779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:16.024014 10779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:16.031183 10779 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.252:38891
I20260808 18:36:16.031445 10852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.252:38891 every 8 connection(s)
I20260808 18:36:16.032159 10779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
I20260808 18:36:16.034844 10853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:16.037637 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10779
I20260808 18:36:16.037806 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786214007390461-28655-0/minicluster-data/master-2/wal/instance
I20260808 18:36:16.038093 28655 external_mini_cluster.cc:1428] Setting key a078cfd9d72a14eb34fefcf748508da3
I20260808 18:36:16.039474 10853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:16.045862 10696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 3
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:36:16.052348 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.041531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.27.251.254:37973)
0808 18:36:16.041656 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.041661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.042084 (+ 423us) server_negotiation.cc:407] Connection header received
0808 18:36:16.042267 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.042272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.042323 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.042413 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.043247 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.043741 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.044823 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.045051 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.045158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.047928 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.047931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.047933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.047977 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.049899 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.050542 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.050548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.050550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.050730 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.051129 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.051133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.051135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.051378 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.051477 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.052089 (+ 612us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.052187 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:36:16.053308 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.041440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:37973 (local address 127.0.0.1:60084)
0808 18:36:16.041947 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:16.042000 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:16.042143 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:16.042480 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:16.042489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:16.042843 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:16.043112 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.043123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.043870 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.043874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.044699 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.044709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.045208 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.045212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.045362 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.045898 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:16.045916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:16.047775 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:16.050024 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.050037 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.050045 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.050448 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.050839 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.050842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.050844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.051038 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.051537 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:16.051543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:16.051834 (+ 291us) client_negotiation.cc:769] Sending connection context
0808 18:36:16.052005 (+ 171us) client_negotiation.cc:240] Negotiation successful
0808 18:36:16.052307 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260808 18:36:16.053567 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.039379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.27.251.252:38891)
0808 18:36:16.039797 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.039817 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.039862 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:36:16.040681 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.040702 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.041162 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.041491 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.042516 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.043534 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.044376 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.044866 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.045045 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.048496 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.048529 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.048544 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.048579 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.050550 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.051049 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.051057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.051061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.051158 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.051516 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.051520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.051521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.051819 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.052010 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.052385 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.052568 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260808 18:36:16.055256 10853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:16.056825 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.046286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48806)
0808 18:36:16.046451 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:16.046465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:16.046546 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:16.047012 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:16.047016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:16.047193 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:16.047294 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.047302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.048467 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.048471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.049236 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.049250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.049938 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.049942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.050028 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.050539 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:16.050553 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:16.051922 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:16.055146 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.055148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.055149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.055412 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.055730 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.055733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.055735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.055832 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.056299 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:16.056302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:16.056398 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 18:36:16.056509 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:36:16.056669 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260808 18:36:16.057636 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.046364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.27.251.252:38891)
0808 18:36:16.046753 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.046776 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.046804 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:36:16.046846 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.046848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.046880 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.046951 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.047467 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.048305 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.049411 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.049800 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.049917 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.052823 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.052826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.052829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.052851 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.054738 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.055528 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.055532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.055533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.055619 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.055946 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.055949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.055951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.056165 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.056312 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.057338 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.057464 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260808 18:36:16.058934 10773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } attempt: 3
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:16.061300 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: Bootstrap starting.
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:36:16.064242 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: Neither blocks nor log segments found. Creating new log.
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:36:16.065253 10696 log.cc:824] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:16.066533 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.056022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.27.251.253:46109)
0808 18:36:16.056128 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.056133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.056218 (+ 85us) server_negotiation.cc:407] Connection header received
0808 18:36:16.056413 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.056424 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.056537 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.056610 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.057142 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.057939 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.059201 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.059630 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.059729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.062333 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.062340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.062347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.062376 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.064291 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.064960 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.064963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.064965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.065045 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.065390 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.065394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.065397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.065758 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.065828 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.066239 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.066364 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 18:36:16.067109 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.055740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:46109 (local address 127.0.0.1:43694)
0808 18:36:16.056155 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:16.056169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:16.056274 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:16.056735 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:16.056738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:16.056904 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:16.057002 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.057008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.058081 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.058084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.058799 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.058807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.059902 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.059908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.060018 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.060495 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:16.060536 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:16.062144 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:16.064501 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.064540 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.064545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.064848 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.065170 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.065173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.065175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.065274 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.065909 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:16.065912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:16.066014 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 18:36:16.066747 (+ 733us) client_negotiation.cc:240] Negotiation successful
0808 18:36:16.066929 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260808 18:36:16.067346 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.055949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.27.251.254:37973)
0808 18:36:16.056055 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.056059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.056239 (+ 180us) server_negotiation.cc:407] Connection header received
0808 18:36:16.056372 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.056375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.056419 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.056527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.057110 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.057586 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.058454 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.058762 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.058859 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.062387 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.062390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.062393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.062416 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.064799 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.065566 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.065570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.065571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.065669 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.066282 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.066285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.066286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.066501 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.066669 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.067069 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.067188 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:36:16.067657 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: No bootstrap required, opened a new log
I20260808 18:36:16.067788 10853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:36:16.069276 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.059332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48810)
0808 18:36:16.059594 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:16.059607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:16.059678 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:16.060158 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:16.060165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:16.060359 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:16.060452 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.060459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.061484 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.061487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.062224 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.062235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.062861 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.062865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.062943 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.063375 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:16.063390 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:16.064905 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:16.067133 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.067136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.067138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.067383 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.068085 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.068090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.068092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.068198 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.068708 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:16.068711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:16.068835 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 18:36:16.068951 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 18:36:16.069098 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260808 18:36:16.069312 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.059815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.27.251.252:38891)
0808 18:36:16.059930 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.059933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.059945 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:36:16.059976 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.059978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.060028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.060591 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.060676 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.061350 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.062375 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.062728 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.063157 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.065070 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.065076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.065082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.065111 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.067009 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.067866 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.067869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.067871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.067958 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.068331 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.068334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.068335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.068573 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.068712 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.069024 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.069158 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:36:16.072250 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: Bootstrap starting.
I20260808 18:36:16.072801 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.073050 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:16.073124 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4b0de1ed914b3d9861bb45f320c7ae, State: Initialized, Role: FOLLOWER
I20260808 18:36:16.073629 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:36:16.074211 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.074429 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: This master's current role is: FOLLOWER
W20260808 18:36:16.074936 10646 tablet.cc:2369] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:16.075093 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:16.075198 10773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:16.076032 10773 log.cc:824] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:16.078320 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: No bootstrap required, opened a new log
I20260808 18:36:16.078348 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.068346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.27.251.252:38891)
0808 18:36:16.068445 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.068449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.068719 (+ 270us) server_negotiation.cc:407] Connection header received
0808 18:36:16.068871 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.068874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.068930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.069001 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.069882 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.070589 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.071791 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.072108 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.072216 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.074540 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.074544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.074547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.074569 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.076675 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.077173 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.077176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.077178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.077251 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.077503 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.077506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.077508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.077702 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.077775 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.078094 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.078207 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:36:16.078756 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.068273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38891 (local address 127.0.0.1:48816)
0808 18:36:16.068648 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:16.068663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:16.068772 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:16.069097 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:16.069100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:16.069273 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:16.069787 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.069795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.070718 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.070721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.071653 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:16.071664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.072237 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.072241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:16.072342 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.072886 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:16.072900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:16.074406 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:16.076791 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.076797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.076803 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.077068 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.077343 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:16.077345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:16.077347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:16.077419 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:16.077834 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:16.077837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:16.077925 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 18:36:16.078475 (+ 550us) client_negotiation.cc:240] Negotiation successful
0808 18:36:16.078605 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":184,"threads_started":1}
I20260808 18:36:16.082033 10853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: Bootstrap starting.
Aug 08 18:36:16 dist-test-slave-fd7l krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
W20260808 18:36:16.083967 10873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:36:16.084046 10873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:36:16.080649 10625 master_runner.cc:186] Error getting master registration for 127.27.251.253:46109: OK, instance_id { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" instance_seqno: 1786214175805330 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:36:16.084659 10625 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:36:16.087528 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.069686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.27.251.253:46109)
0808 18:36:16.069783 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.069787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.076721 (+ 6934us) server_negotiation.cc:407] Connection header received
0808 18:36:16.076910 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.076914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.076961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.077040 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.077963 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.078658 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.080017 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.080364 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.080467 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.083606 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.083610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.083612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.083637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.085348 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.085878 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.085881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.085883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.085985 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.086288 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.086291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.086292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.086494 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.086666 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.087271 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.087382 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:36:16.088027 10773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.088240 10773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:16.088290 10773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04696b8b0ac64507b0393710a116dee1, State: Initialized, Role: FOLLOWER
I20260808 18:36:16.088840 10773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.089375 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.089775 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.090885 10853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:16.091429 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:36:16.091876 10853 log.cc:824] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:16.092828 10773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:16.094254 10853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: No bootstrap required, opened a new log
W20260808 18:36:16.097884 10892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:36:16.098009 10892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:36:16.099362 10853 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.099670 10853 raft_consensus.cc:374] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:16.099738 10853 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05b0c7bd8404c1099d6f70c81cf19bb, State: Initialized, Role: FOLLOWER
I20260808 18:36:16.100178 10853 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.101666 10853 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:16.100286 10704 master_runner.cc:186] Error getting master registration for 127.27.251.252:38891: OK, instance_id { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" instance_seqno: 1786214176029225 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:36:16.103796 10704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:36:16.109336 10779 master_runner.cc:263] No leader master found from master c05b0c7bd8404c1099d6f70c81cf19bb
I20260808 18:36:16.109411 10779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:36:16.113040 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.098632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.27.251.252:38891)
0808 18:36:16.098743 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.098747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.103303 (+ 4556us) server_negotiation.cc:407] Connection header received
0808 18:36:16.103531 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.103534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.103581 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.103641 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.104852 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.105608 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.106807 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.107159 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.107241 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.109376 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.109379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.109382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.109406 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.111207 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.111697 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.111701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.111703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.111785 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.112080 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.112083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.112085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.112291 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.112410 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.112761 (+ 351us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.112871 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:36:16.116178 10861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:16.116374 10861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.117476 10861 leader_election.cc:288] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04696b8b0ac64507b0393710a116dee1 (127.27.251.253:46109), c05b0c7bd8404c1099d6f70c81cf19bb (127.27.251.252:38891)
I20260808 18:36:16.118054 10747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04696b8b0ac64507b0393710a116dee1" is_pre_election: true
I20260808 18:36:16.118343 10747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4b0de1ed914b3d9861bb45f320c7ae in term 0.
I20260808 18:36:16.118745 10826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" is_pre_election: true
I20260808 18:36:16.119028 10826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee4b0de1ed914b3d9861bb45f320c7ae in term 0.
I20260808 18:36:16.119490 10641 leader_election.cc:302] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05b0c7bd8404c1099d6f70c81cf19bb, ee4b0de1ed914b3d9861bb45f320c7ae; no voters:
I20260808 18:36:16.119796 10861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:36:16.119895 10861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:36:16.119951 10861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:16.120260 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.120388 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.121210 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.099104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.27.251.254:37973)
0808 18:36:16.099215 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.099219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.105038 (+ 5819us) server_negotiation.cc:407] Connection header received
0808 18:36:16.105200 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.105204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.105248 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.105294 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.106200 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.106949 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.108038 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.108413 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.108540 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.117563 (+ 9023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.117568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.117570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.117599 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.119189 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.119712 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.119715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.119717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.119804 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.120101 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.120106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.120108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.120398 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.120574 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.120916 (+ 342us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.121056 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":47}
I20260808 18:36:16.121461 10861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.121870 10861 leader_election.cc:288] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [CANDIDATE]: Term 1 election: Requested vote from peers 04696b8b0ac64507b0393710a116dee1 (127.27.251.253:46109), c05b0c7bd8404c1099d6f70c81cf19bb (127.27.251.252:38891)
I20260808 18:36:16.122731 10747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04696b8b0ac64507b0393710a116dee1"
I20260808 18:36:16.122867 10747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:16.124207 10747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4b0de1ed914b3d9861bb45f320c7ae in term 1.
I20260808 18:36:16.124569 10642 leader_election.cc:302] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04696b8b0ac64507b0393710a116dee1, ee4b0de1ed914b3d9861bb45f320c7ae; no voters:
I20260808 18:36:16.124722 10826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb"
I20260808 18:36:16.124846 10826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:16.126082 10826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4b0de1ed914b3d9861bb45f320c7ae in term 1.
I20260808 18:36:16.126524 10861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:16.126775 10861 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [term 1 LEADER]: Becoming Leader. State: Replica: ee4b0de1ed914b3d9861bb45f320c7ae, State: Running, Role: LEADER
I20260808 18:36:16.127121 10861 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } }
I20260808 18:36:16.129020 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4b0de1ed914b3d9861bb45f320c7ae. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.129132 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: This master's current role is: LEADER
W20260808 18:36:16.129616 10909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:36:16.129698 10909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:36:16.131052 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:16.100163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.27.251.253:46109)
0808 18:36:16.100261 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:16.100265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:16.109210 (+ 8945us) server_negotiation.cc:407] Connection header received
0808 18:36:16.109390 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:16.109394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:16.109440 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:16.109515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:16.110407 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.111157 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.112367 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:16.112729 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:16.112808 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:16.115047 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:16.115051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:16.115054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:16.115079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:16.117114 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.129594 (+ 12480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.129598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.129600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.129753 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:16.130095 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:16.130099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:16.130101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:16.130335 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:16.130484 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:16.130755 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 18:36:16.130888 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:36:16.134024 10913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:16.134954 10913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:16.143157 10826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [term 1 FOLLOWER]: Refusing update from remote peer ee4b0de1ed914b3d9861bb45f320c7ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:16.143934 10910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:16.150139 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4b0de1ed914b3d9861bb45f320c7ae. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.150259 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.150727 10747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [term 1 FOLLOWER]: Refusing update from remote peer ee4b0de1ed914b3d9861bb45f320c7ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:16.151381 10861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:16.154948 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4b0de1ed914b3d9861bb45f320c7ae. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.155058 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.160176 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.160305 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:16.161368 10898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.161481 10898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.161960 10914 mvcc.cc:205] Tried to move back new op lower bound from 7316333265474412544 to 7316333265420161024. Current Snapshot: MvccSnapshot[applied={T|T < 7316333265474412544}]
I20260808 18:36:16.162410 10915 mvcc.cc:205] Tried to move back new op lower bound from 7316333265474412544 to 7316333265420161024. Current Snapshot: MvccSnapshot[applied={T|T < 7316333265474412544}]
I20260808 18:36:16.166051 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.166162 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:16.169446 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b0de1ed914b3d9861bb45f320c7ae" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37973 } } peers { permanent_uuid: "04696b8b0ac64507b0393710a116dee1" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 46109 } } peers { permanent_uuid: "c05b0c7bd8404c1099d6f70c81cf19bb" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38891 } } }
I20260808 18:36:16.169554 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:36:16.170277 10913 catalog_manager.cc:1347] Generated new cluster ID: afcf71bd498c4e45aa52a0a38a59ed30
I20260808 18:36:16.170346 10913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:16.170403 10917 mvcc.cc:205] Tried to move back new op lower bound from 7316333265474412544 to 7316333265420161024. Current Snapshot: MvccSnapshot[applied={T|T < 7316333265474412544}]
I20260808 18:36:16.197374 10913 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:36:16.198781 10913 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:16.220630 10913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee4b0de1ed914b3d9861bb45f320c7ae: Generated new TSK 0
I20260808 18:36:16.221596 10913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:36:17.099466 10892 catalog_manager.cc:1259] Loaded cluster ID: afcf71bd498c4e45aa52a0a38a59ed30
I20260808 18:36:17.099579 10892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: loading cluster ID for follower catalog manager: success
I20260808 18:36:17.101498 10892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: acquiring CA information for follower catalog manager: success
I20260808 18:36:17.102831 10892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04696b8b0ac64507b0393710a116dee1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:36:17.131099 10909 catalog_manager.cc:1259] Loaded cluster ID: afcf71bd498c4e45aa52a0a38a59ed30
I20260808 18:36:17.131186 10909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: loading cluster ID for follower catalog manager: success
I20260808 18:36:17.132980 10909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: acquiring CA information for follower catalog manager: success
I20260808 18:36:17.134181 10909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c05b0c7bd8404c1099d6f70c81cf19bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:36:17.284761 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10625
I20260808 18:36:17.294158 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10704
I20260808 18:36:17.303645 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10779
2026-08-08T18:36:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3204 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:36:17 dist-test-slave-fd7l krb5kdc[10930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:36:17 dist-test-slave-fd7l krb5kdc[10930](info): set up 2 sockets
Aug 08 18:36:17 dist-test-slave-fd7l krb5kdc[10930](info): commencing operation
krb5kdc: starting...
W20260808 18:36:18.423184 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[10930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:36:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:36:18.559024 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35265
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:18.678134 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:18.678386 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:18.678426 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:18.682473 10943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:18.682533 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:18.682552 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:18.682569 10943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:18.682608 10943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:18.687925 10943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:18.689018 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:18.690070 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:18.698971 10951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:18.699071 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:18.699239 10943 server_base.cc:1034] running on GCE node
W20260808 18:36:18.698956 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:18.699941 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:18.701256 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:18.702530 10943 hybrid_clock.cc:648] HybridClock initialized: now 1786214178702491 us; error 48 us; skew 500 ppm
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[10930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214178, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:18.705498 10943 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:18.706332 10943 webserver.cc:460] Webserver started at http://127.27.251.254:43391/ using document root <none> and password file <none>
I20260808 18:36:18.706881 10943 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:18.706941 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:18.707109 10943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:18.709230 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "55298c0f3d3546328eb44744a262fd0f"
format_stamp: "Formatted at 2026-08-08 18:36:18 on dist-test-slave-fd7l"
server_key: "74c198682b8214a115d2ecd54b02823f"
server_key_iv: "39dbff687d7c4e0e488db4708df6be99"
server_key_version: "encryptionkey@0"
I20260808 18:36:18.709867 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "55298c0f3d3546328eb44744a262fd0f"
format_stamp: "Formatted at 2026-08-08 18:36:18 on dist-test-slave-fd7l"
server_key: "74c198682b8214a115d2ecd54b02823f"
server_key_iv: "39dbff687d7c4e0e488db4708df6be99"
server_key_version: "encryptionkey@0"
I20260808 18:36:18.713620 10943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:36:18.716143 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:18.717532 10943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:18.717646 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "55298c0f3d3546328eb44744a262fd0f"
format_stamp: "Formatted at 2026-08-08 18:36:18 on dist-test-slave-fd7l"
server_key: "74c198682b8214a115d2ecd54b02823f"
server_key_iv: "39dbff687d7c4e0e488db4708df6be99"
server_key_version: "encryptionkey@0"
I20260808 18:36:18.717732 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:18.728434 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:18.731747 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:18.731954 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:18.739318 10943 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35265
I20260808 18:36:18.739342 11013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35265 every 8 connection(s)
I20260808 18:36:18.740272 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:18.743152 11014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:18.747334 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 10943
I20260808 18:36:18.747447 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:36:18.747777 28655 external_mini_cluster.cc:1428] Setting key 5eebb24201a83e8b3ff8c6ff6128a815
I20260808 18:36:18.749116 11014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: Bootstrap starting.
I20260808 18:36:18.751900 11014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:18.752841 11014 log.cc:824] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:18.754925 11014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: No bootstrap required, opened a new log
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[10930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:18.759274 11014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } }
I20260808 18:36:18.759557 11014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:18.759608 11014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55298c0f3d3546328eb44744a262fd0f, State: Initialized, Role: FOLLOWER
I20260808 18:36:18.760180 11014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } }
I20260808 18:36:18.760342 11014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:18.760430 11014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:18.760551 11014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:18.761722 11014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } }
I20260808 18:36:18.762113 11014 leader_election.cc:302] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55298c0f3d3546328eb44744a262fd0f; no voters:
I20260808 18:36:18.762502 11014 leader_election.cc:288] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:36:18.762629 11019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:18.762887 11019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [term 1 LEADER]: Becoming Leader. State: Replica: 55298c0f3d3546328eb44744a262fd0f, State: Running, Role: LEADER
I20260808 18:36:18.763211 11019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } }
I20260808 18:36:18.763592 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:18.749319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.27.251.254:35265)
0808 18:36:18.749732 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:18.749752 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:18.749795 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:36:18.750670 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:18.750697 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:18.751099 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:18.751453 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:18.752214 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:18.753670 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:18.754549 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:18.755245 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:18.755414 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:18.758036 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:18.758042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:18.758059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:18.758090 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:18.760108 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:18.760642 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:18.760648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:18.760654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:18.760805 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:18.761094 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:18.761097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:18.761098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:18.761660 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:18.761874 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:18.762241 (+ 367us) server_negotiation.cc:299] Negotiation successful
0808 18:36:18.762573 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260808 18:36:18.763777 11014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:18.765981 11020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55298c0f3d3546328eb44744a262fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } } }
I20260808 18:36:18.766142 11020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:18.766453 11021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55298c0f3d3546328eb44744a262fd0f. Latest consensus state: current_term: 1 leader_uuid: "55298c0f3d3546328eb44744a262fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55298c0f3d3546328eb44744a262fd0f" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35265 } } }
I20260808 18:36:18.766552 11021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:18.766621 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 18:36:18.769717 11035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:36:18.769801 11035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:36:18.769919 11032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:18.770624 11032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:18.775372 11032 catalog_manager.cc:1347] Generated new cluster ID: e8a9df5ae3f34157a0ee366d7ed340d4
I20260808 18:36:18.775441 11032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:18.787518 11032 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:36:18.788551 11032 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:18.797591 11032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55298c0f3d3546328eb44744a262fd0f: Generated new TSK 0
I20260808 18:36:18.798296 11032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:36:18.822760 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 10943
2026-08-08T18:36:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1517 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[11042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[11042](info): set up 2 sockets
Aug 08 18:36:18 dist-test-slave-fd7l krb5kdc[11042](info): commencing operation
krb5kdc: starting...
W20260808 18:36:19.898936 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:36:19 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:36:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:36:20.027786 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:20.140317 11056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:20.140602 11056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:20.140642 11056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:20.144629 11056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:20.144687 11056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:20.144706 11056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:20.144723 11056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:20.144738 11056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:20.150120 11056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:20.151175 11056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:20.152261 11056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:20.160890 11061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.160959 11062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:20.161100 11056 server_base.cc:1034] running on GCE node
W20260808 18:36:20.161158 11064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:20.161744 11056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:20.162914 11056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:20.164106 11056 hybrid_clock.cc:648] HybridClock initialized: now 1786214180164088 us; error 29 us; skew 500 ppm
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:20.171533 11056 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:20.172443 11056 webserver.cc:460] Webserver started at http://127.27.251.254:33629/ using document root <none> and password file <none>
I20260808 18:36:20.173048 11056 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:20.173112 11056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:20.173283 11056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:20.175426 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "ed97245b92d348bf8cd2184c724fe23d"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "1543185eb856b17ce4b25c72084785b9"
server_key_iv: "dd55de814a2cc0884b952f6118f994c2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.175954 11056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "ed97245b92d348bf8cd2184c724fe23d"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "1543185eb856b17ce4b25c72084785b9"
server_key_iv: "dd55de814a2cc0884b952f6118f994c2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.179733 11056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 18:36:20.182339 11071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.183656 11056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:20.183794 11056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "ed97245b92d348bf8cd2184c724fe23d"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "1543185eb856b17ce4b25c72084785b9"
server_key_iv: "dd55de814a2cc0884b952f6118f994c2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.183921 11056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:20.209746 11056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:20.212937 11056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:20.213152 11056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:20.220858 11056 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:41795
I20260808 18:36:20.221172 11123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:41795 every 8 connection(s)
I20260808 18:36:20.221899 11056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:20.224821 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11056
I20260808 18:36:20.224843 11124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:20.224996 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:36:20.225314 28655 external_mini_cluster.cc:1428] Setting key 3f693274927c9b56ce987658226daf93
I20260808 18:36:20.232015 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap starting.
I20260808 18:36:20.235077 11124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Neither blocks nor log segments found. Creating new log.
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:20.235896 11124 log.cc:824] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:20.238093 11124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: No bootstrap required, opened a new log
I20260808 18:36:20.241199 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.226808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.27.251.254:41795)
0808 18:36:20.227311 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:20.227328 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:20.227368 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:36:20.228144 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:20.228163 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:20.228614 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:20.228940 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:20.229972 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.231220 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.232423 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.232994 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.233123 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:20.236136 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:20.236141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:20.236156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:20.236187 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:20.238091 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.238676 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.238681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.238686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.238804 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.239107 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.239109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.239110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.239517 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:20.239653 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:20.239988 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 18:36:20.240310 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":188,"threads_started":1}
I20260808 18:36:20.242622 11124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:20.242918 11124 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:20.242969 11124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Initialized, Role: FOLLOWER
I20260808 18:36:20.243427 11124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:20.243588 11124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:20.243662 11124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:20.243777 11124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:20.244818 11124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:20.245160 11124 leader_election.cc:302] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed97245b92d348bf8cd2184c724fe23d; no voters:
I20260808 18:36:20.245473 11124 leader_election.cc:288] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:36:20.246223 11129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:20.246629 11129 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 LEADER]: Becoming Leader. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Running, Role: LEADER
I20260808 18:36:20.247072 11129 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:20.247426 11124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:20.248389 11129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed97245b92d348bf8cd2184c724fe23d. Latest consensus state: current_term: 1 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:20.248571 11129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:20.248885 11130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:20.248993 11130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
W20260808 18:36:20.252846 11146 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:36:20.252935 11146 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:36:20.253006 11139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:20.253726 11139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:20.260010 11139 catalog_manager.cc:1347] Generated new cluster ID: 44a2ea6d0f2c4e228996f8d8a3e512a1
I20260808 18:36:20.260102 11139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:20.276830 11139 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:36:20.277933 11139 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:20.282969 11139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Generated new TSK 0
I20260808 18:36:20.283813 11139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:36:20.323210 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:20.440295 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:20.440577 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:20.440620 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:20.444540 11152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:20.444593 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:20.444649 11152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:20.450284 11152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:20.451318 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:20.452304 11152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:20.462028 11157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.462010 11160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.462028 11158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:20.462278 11152 server_base.cc:1034] running on GCE node
I20260808 18:36:20.462782 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:20.463475 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:20.464661 11152 hybrid_clock.cc:648] HybridClock initialized: now 1786214180464647 us; error 23 us; skew 500 ppm
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:20.467540 11152 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:20.468376 11152 webserver.cc:460] Webserver started at http://127.27.251.193:40831/ using document root <none> and password file <none>
I20260808 18:36:20.469028 11152 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:20.469102 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:20.469276 11152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:20.471266 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "c314e40244c647968079c60718c0c802"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "5ab0ab0578c313e26b225fbf78ba873f"
server_key_iv: "fe5c2514f2e3418403edf57376d30939"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.471792 11152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "c314e40244c647968079c60718c0c802"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "5ab0ab0578c313e26b225fbf78ba873f"
server_key_iv: "fe5c2514f2e3418403edf57376d30939"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.475533 11152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:36:20.478021 11167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.479285 11152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:36:20.479414 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c314e40244c647968079c60718c0c802"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "5ab0ab0578c313e26b225fbf78ba873f"
server_key_iv: "fe5c2514f2e3418403edf57376d30939"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.479527 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:20.501891 11152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:20.504820 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:20.505019 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:20.505635 11152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:20.506464 11152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:20.506517 11152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.506582 11152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:20.506611 11152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.515515 11152 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:41825
I20260808 18:36:20.515542 11281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:41825 every 8 connection(s)
I20260808 18:36:20.516397 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:20.520871 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11152
I20260808 18:36:20.521018 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:36:20.521308 28655 external_mini_cluster.cc:1428] Setting key 709a812f52e939c8410875955290ad15
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:20.530849 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.518251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:47709 (local address 127.27.251.254:41795)
0808 18:36:20.518392 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:20.518397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:20.519166 (+ 769us) server_negotiation.cc:407] Connection header received
0808 18:36:20.520710 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:20.520713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:20.520776 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:20.520896 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:20.522201 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.522746 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.523953 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.524275 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.524385 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:20.527154 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:20.527158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:20.527161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:20.527192 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:20.528768 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.529294 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.529298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.529300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.529384 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.529748 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.529751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.529753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.529932 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:20.530050 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:20.530550 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 18:36:20.530677 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:36:20.531654 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.518560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:47709)
0808 18:36:20.519008 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:20.519051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:20.520376 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:20.521108 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:20.521120 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:20.521559 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:20.522034 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.522047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.522877 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.522880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:20.523822 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.523832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.524437 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.524442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:20.524670 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:20.525301 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:20.525327 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:20.526993 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:20.528902 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:20.528908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:20.528922 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:20.529185 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:20.529480 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:20.529485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:20.529491 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:20.529644 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:20.530050 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:20.530057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:20.530307 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 18:36:20.530503 (+ 196us) client_negotiation.cc:240] Negotiation successful
0808 18:36:20.530824 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:36:20.533124 11282 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:20.533394 11282 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:20.533906 11282 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:36:20.535725 11088 ts_manager.cc:194] Registered new tserver with Master: c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:20.536990 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:47709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:36:20.552217 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:20.669816 11288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:20.670087 11288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:20.670131 11288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:20.674307 11288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:20.674368 11288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:20.674458 11288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:20.679868 11288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:20.681010 11288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:20.682191 11288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:20.692827 11294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.692837 11296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.692862 11293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:20.693045 11288 server_base.cc:1034] running on GCE node
I20260808 18:36:20.693611 11288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:20.694341 11288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:20.695576 11288 hybrid_clock.cc:648] HybridClock initialized: now 1786214180695558 us; error 33 us; skew 500 ppm
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:20.698634 11288 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:20.699558 11288 webserver.cc:460] Webserver started at http://127.27.251.194:44937/ using document root <none> and password file <none>
I20260808 18:36:20.700147 11288 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:20.700242 11288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:20.700502 11288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:20.702665 11288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "9141e7d544a7c72950cd31323b21a9de"
server_key_iv: "a36b05f1fe0f0b59e5287ae9ee69ebc2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.703240 11288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "9141e7d544a7c72950cd31323b21a9de"
server_key_iv: "a36b05f1fe0f0b59e5287ae9ee69ebc2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.706962 11288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260808 18:36:20.709347 11303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.710574 11288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:20.710718 11288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "9141e7d544a7c72950cd31323b21a9de"
server_key_iv: "a36b05f1fe0f0b59e5287ae9ee69ebc2"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.710826 11288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:20.729609 11288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:20.732594 11288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:20.732790 11288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:20.733419 11288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:20.734216 11288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:20.734268 11288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.734306 11288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:20.734328 11288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.743099 11288 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:46513
I20260808 18:36:20.743162 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:46513 every 8 connection(s)
I20260808 18:36:20.744060 11288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:36:20.753386 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11288
I20260808 18:36:20.753523 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:36:20.753885 28655 external_mini_cluster.cc:1428] Setting key bb6bcdff6e8ded037ae71b18110b83f4
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:20.758420 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.745870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:47763 (local address 127.27.251.254:41795)
0808 18:36:20.746027 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:20.746033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:20.746887 (+ 854us) server_negotiation.cc:407] Connection header received
0808 18:36:20.748605 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:20.748609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:20.748666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:20.748750 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:20.750101 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.750645 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.751761 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.751971 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.752048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:20.754631 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:20.754634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:20.754636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:20.754660 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:20.756177 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.756775 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.756778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.756780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.756863 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:20.757216 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:20.757221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:20.757222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:20.757399 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:20.757517 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:20.758114 (+ 597us) server_negotiation.cc:299] Negotiation successful
0808 18:36:20.758253 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:36:20.759485 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.746298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:47763)
0808 18:36:20.746739 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:20.746781 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:20.748235 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:20.748955 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:20.748962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:20.749357 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:20.749942 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.749955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.750766 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.750768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:20.751654 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.751663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.752104 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.752106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:20.752243 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:20.752814 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:20.752830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:20.754393 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:20.756302 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:20.756308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:20.756323 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:20.756669 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:20.756957 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:20.756962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:20.756964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:20.757117 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:20.757509 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:20.757516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:20.757835 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 18:36:20.758069 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 18:36:20.758390 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260808 18:36:20.760758 11420 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:20.761040 11420 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:20.761631 11420 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
I20260808 18:36:20.762861 11088 ts_manager.cc:194] Registered new tserver with Master: e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
I20260808 18:36:20.763466 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:47763
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:36:20.783596 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:20.898307 11426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:20.898587 11426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:20.898790 11426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:20.903496 11426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:20.903579 11426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:20.903682 11426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:20.908951 11426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:20.909987 11426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:20.910990 11426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:20.920747 11433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.920785 11436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:20.920769 11434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:20.920840 11426 server_base.cc:1034] running on GCE node
I20260808 18:36:20.921375 11426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:20.921988 11426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:20.923177 11426 hybrid_clock.cc:648] HybridClock initialized: now 1786214180923151 us; error 46 us; skew 500 ppm
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:20.925907 11426 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:20.926733 11426 webserver.cc:460] Webserver started at http://127.27.251.195:44385/ using document root <none> and password file <none>
I20260808 18:36:20.927290 11426 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:20.927382 11426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:20.927618 11426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:20.929704 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "5e60d8245560480c8c80044501e178fc"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "406cacffb74639e2f8809250c13c5515"
server_key_iv: "1726bbd16811e38896deace1541d5292"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.930298 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "5e60d8245560480c8c80044501e178fc"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "406cacffb74639e2f8809250c13c5515"
server_key_iv: "1726bbd16811e38896deace1541d5292"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.933974 11426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:20.936385 11443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.937691 11426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:20.937824 11426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "5e60d8245560480c8c80044501e178fc"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "406cacffb74639e2f8809250c13c5515"
server_key_iv: "1726bbd16811e38896deace1541d5292"
server_key_version: "encryptionkey@0"
I20260808 18:36:20.937930 11426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:20.977222 11426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:20.980157 11426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:20.980368 11426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:20.981022 11426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:20.981858 11426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:20.981928 11426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.982000 11426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:20.982062 11426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:20.991328 11426 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:46373
I20260808 18:36:20.991359 11562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:46373 every 8 connection(s)
I20260808 18:36:20.992215 11426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:21.000841 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11426
I20260808 18:36:21.000998 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:36:21.001286 28655 external_mini_cluster.cc:1428] Setting key 6a4686d59d6c13c8d2aab87aeb167f3f
Aug 08 18:36:20 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214180, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:21.006032 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.994076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:58701 (local address 127.27.251.254:41795)
0808 18:36:20.994213 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:20.994217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:20.995103 (+ 886us) server_negotiation.cc:407] Connection header received
0808 18:36:20.996492 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:20.996496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:20.996570 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:20.996686 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:20.997835 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.998371 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.999510 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.999744 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.999858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.002592 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:21.002595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:21.002598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:21.002622 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:21.004172 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.004734 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.004737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.004738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.004813 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.005104 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.005106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.005107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.005270 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:21.005356 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.005827 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.005922 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:36:21.006807 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:20.994469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:58701)
0808 18:36:20.994950 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:20.995008 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:20.996273 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:20.996859 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:20.996866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:20.997233 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:20.997690 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.997702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.998502 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.998506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:20.999383 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:20.999393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:20.999899 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:20.999903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.000081 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.000702 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:21.000719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:21.002379 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:21.004277 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.004282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.004293 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.004647 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.004892 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.004894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.004895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.005024 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.005357 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:21.005361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:21.005575 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.005756 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.006000 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260808 18:36:21.007965 11563 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:21.008234 11563 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:21.008750 11563 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
I20260808 18:36:21.009766 11088 ts_manager.cc:194] Registered new tserver with Master: 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373)
I20260808 18:36:21.010280 11088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:58701
I20260808 18:36:21.015285 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:36:21.024173 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.017417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.27.251.254:41795)
0808 18:36:21.017557 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.017562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.017771 (+ 209us) server_negotiation.cc:407] Connection header received
0808 18:36:21.017856 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.017858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.017893 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.017958 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:21.018636 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.019109 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.019904 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.020108 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.020181 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.021512 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:21.021515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:21.021517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:21.021542 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:21.022782 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.023181 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.023184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.023185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.023253 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.023479 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.023481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.023482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.023686 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:21.023786 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.023983 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.024070 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:36:21.026742 11088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:36:21.030468 11088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:36:21.045523 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.039720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.0.0.1:60844)
0808 18:36:21.040570 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.040584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.040680 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.041277 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.041280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.041298 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:21.041380 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.041392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042861 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.042864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.044029 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.044040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.044969 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.045069 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.045092 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.045191 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.045351 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":49,"thread_start_us":88,"threads_started":1}
I20260808 18:36:21.045534 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.039733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.27.251.194:46513)
0808 18:36:21.040235 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.040249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.040273 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:36:21.040354 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.040359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.040617 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.040774 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:21.041083 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042305 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044123 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.044759 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044863 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.045013 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.045234 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.045403 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":119,"threads_started":1}
I20260808 18:36:21.045527 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.039375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:46513 (local address 127.0.0.1:55342)
0808 18:36:21.039995 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.040008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.040088 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.040761 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.040768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.040797 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:21.040952 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.040959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042442 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.042445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.043993 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.043999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.044886 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.045024 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.045064 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.045183 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.045360 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":245,"thread_start_us":105,"threads_started":1}
I20260808 18:36:21.046141 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.039958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.27.251.193:41825)
0808 18:36:21.040300 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.040308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.040779 (+ 471us) server_negotiation.cc:407] Connection header received
0808 18:36:21.040921 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.040928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.041135 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.041281 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:21.041558 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042724 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044182 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.044838 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.044973 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.045121 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.045886 (+ 765us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.045999 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":128,"threads_started":1}
I20260808 18:36:21.046856 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.038978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46373 (local address 127.0.0.1:45846)
0808 18:36:21.039357 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.039408 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.039930 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.040869 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.040872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.040885 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:21.040959 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.040964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042322 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.042328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.043984 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.043996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.046467 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.046471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.046567 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.046590 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.046674 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.046737 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":39,"thread_start_us":97,"threads_started":1}
I20260808 18:36:21.048630 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.039126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.27.251.195:46373)
0808 18:36:21.040352 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.040363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.040391 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:36:21.040489 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.040494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.040724 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.041111 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:21.041271 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.042196 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.045082 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.045942 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.046482 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.046605 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.046695 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.046780 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1126,"thread_start_us":191,"threads_started":1}
I20260808 18:36:21.049268 11216 tablet_service.cc:1467] Processing CreateTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (DEFAULT_TABLE table=test-table [id=2aa254b349cf486b9013fecbd524212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:21.049268 11352 tablet_service.cc:1467] Processing CreateTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (DEFAULT_TABLE table=test-table [id=2aa254b349cf486b9013fecbd524212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:21.050494 11216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:21.050494 11352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:21.051219 11493 tablet_service.cc:1467] Processing CreateTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (DEFAULT_TABLE table=test-table [id=2aa254b349cf486b9013fecbd524212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:21.052120 11493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:21.057953 11580 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Bootstrap starting.
I20260808 18:36:21.058647 11582 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Bootstrap starting.
I20260808 18:36:21.058826 11581 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Bootstrap starting.
I20260808 18:36:21.060876 11581 tablet_bootstrap.cc:654] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:21.060889 11580 tablet_bootstrap.cc:654] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:21.061317 11582 tablet_bootstrap.cc:654] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:21.061832 11581 log.cc:824] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:21.061833 11580 log.cc:824] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:21.062189 11582 log.cc:824] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:21.063800 11580 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: No bootstrap required, opened a new log
I20260808 18:36:21.063926 11581 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: No bootstrap required, opened a new log
I20260808 18:36:21.063948 11582 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: No bootstrap required, opened a new log
I20260808 18:36:21.064054 11580 ts_tablet_manager.cc:1397] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:36:21.064114 11581 ts_tablet_manager.cc:1397] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:36:21.064123 11582 ts_tablet_manager.cc:1397] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:36:21.067010 11580 raft_consensus.cc:348] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.067283 11580 raft_consensus.cc:374] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:21.067351 11580 raft_consensus.cc:729] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e60d8245560480c8c80044501e178fc, State: Initialized, Role: FOLLOWER
I20260808 18:36:21.067860 11580 consensus_queue.cc:260] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.068048 11582 raft_consensus.cc:348] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.068315 11582 raft_consensus.cc:374] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:21.068362 11582 raft_consensus.cc:729] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56f0e39bb224f5aaf0380719e86f4ff, State: Initialized, Role: FOLLOWER
I20260808 18:36:21.068483 11581 raft_consensus.cc:348] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.068735 11581 raft_consensus.cc:374] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:21.068789 11581 raft_consensus.cc:729] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c314e40244c647968079c60718c0c802, State: Initialized, Role: FOLLOWER
I20260808 18:36:21.068742 11563 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
I20260808 18:36:21.068799 11582 consensus_queue.cc:260] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.069012 11580 ts_tablet_manager.cc:1428] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:36:21.069181 11581 consensus_queue.cc:260] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.069576 11420 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
I20260808 18:36:21.069741 11582 ts_tablet_manager.cc:1428] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:36:21.070050 11581 ts_tablet_manager.cc:1428] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:36:21.070416 11282 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
I20260808 18:36:21.210258 11589 raft_consensus.cc:482] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:21.210480 11589 raft_consensus.cc:504] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.211859 11589 leader_election.cc:288] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:21.216804 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.212148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46373 (local address 127.27.251.194:50667)
0808 18:36:21.212281 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.212296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.212399 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.212802 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.212806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.212830 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:21.213024 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.213034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.214103 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.214108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.215422 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.215433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.216310 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.216313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.216424 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.216446 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.216583 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.216664 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":26,"spinlock_wait_cycles":7552}
I20260808 18:36:21.216815 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.212302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:50667 (local address 127.27.251.195:46373)
0808 18:36:21.212467 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.212472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.212491 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:36:21.212604 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.212607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.212666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.212786 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:21.213235 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.213989 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.215564 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.216195 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.216320 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.216377 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.216643 (+ 266us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.216715 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:36:21.217574 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.212438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.27.251.194:50297)
0808 18:36:21.212759 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.212775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.212885 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.213364 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.213367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.213386 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:21.213513 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.213519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.214577 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.214579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.215724 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.215735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.216848 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.216853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.217185 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.217207 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.217315 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.217407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260808 18:36:21.217564 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.212897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:50297 (local address 127.27.251.193:41825)
0808 18:36:21.213055 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.213063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.213083 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:36:21.213178 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.213183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.213243 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.213330 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:21.213639 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.214448 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.215857 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.216704 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.216858 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.216950 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.217350 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.217438 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:36:21.217770 11516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0549b4e3eb0b4540b1318c0cd4ec6d71" candidate_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e60d8245560480c8c80044501e178fc" is_pre_election: true
I20260808 18:36:21.218120 11516 raft_consensus.cc:2454] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56f0e39bb224f5aaf0380719e86f4ff in term 0.
I20260808 18:36:21.218600 11307 leader_election.cc:302] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff; no voters:
I20260808 18:36:21.218892 11589 raft_consensus.cc:2781] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:36:21.218988 11589 raft_consensus.cc:482] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:36:21.218884 11236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0549b4e3eb0b4540b1318c0cd4ec6d71" candidate_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c314e40244c647968079c60718c0c802" is_pre_election: true
I20260808 18:36:21.219046 11589 raft_consensus.cc:3037] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:21.219209 11236 raft_consensus.cc:2454] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56f0e39bb224f5aaf0380719e86f4ff in term 0.
I20260808 18:36:21.220351 11589 raft_consensus.cc:504] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.220774 11589 leader_election.cc:288] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [CANDIDATE]: Term 1 election: Requested vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:21.221139 11516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0549b4e3eb0b4540b1318c0cd4ec6d71" candidate_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e60d8245560480c8c80044501e178fc"
I20260808 18:36:21.221225 11236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0549b4e3eb0b4540b1318c0cd4ec6d71" candidate_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c314e40244c647968079c60718c0c802"
I20260808 18:36:21.221282 11516 raft_consensus.cc:3037] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:21.221359 11236 raft_consensus.cc:3037] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:21.222369 11516 raft_consensus.cc:2454] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56f0e39bb224f5aaf0380719e86f4ff in term 1.
I20260808 18:36:21.222646 11307 leader_election.cc:302] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff; no voters:
I20260808 18:36:21.222781 11236 raft_consensus.cc:2454] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56f0e39bb224f5aaf0380719e86f4ff in term 1.
I20260808 18:36:21.222834 11589 raft_consensus.cc:2781] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:21.223110 11589 raft_consensus.cc:686] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 1 LEADER]: Becoming Leader. State: Replica: e56f0e39bb224f5aaf0380719e86f4ff, State: Running, Role: LEADER
I20260808 18:36:21.223634 11589 consensus_queue.cc:237] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.226955 11085 catalog_manager.cc:5495] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff reported cstate change: term changed from 0 to 1, leader changed from <none> to e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194). New cstate: current_term: 1 leader_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } health_report { overall_health: HEALTHY } } }
I20260808 18:36:21.239955 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.236640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.27.251.194:46513)
0808 18:36:21.236769 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.236773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.236788 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:36:21.236889 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.236891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.236948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.237059 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:36:21.237282 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.237788 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.238583 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.238788 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.238865 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.239125 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:36:21.239528 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:36:21.239630 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.239698 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.239750 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 18:36:21.243286 11564 tablet.cc:2369] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:36:21.245198 11421 tablet.cc:2369] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:21.246517 11516 raft_consensus.cc:1261] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Refusing update from remote peer e56f0e39bb224f5aaf0380719e86f4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:21.246526 11236 raft_consensus.cc:1261] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Refusing update from remote peer e56f0e39bb224f5aaf0380719e86f4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:21.247257 11589 consensus_queue.cc:1035] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:21.247463 11592 consensus_queue.cc:1035] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:21.257025 11598 mvcc.cc:205] Tried to move back new op lower bound from 7316333286377971712 to 7316333286294654976. Current Snapshot: MvccSnapshot[applied={T|T < 7316333286377971712}]
I20260808 18:36:21.258940 11597 mvcc.cc:205] Tried to move back new op lower bound from 7316333286377971712 to 7316333286294654976. Current Snapshot: MvccSnapshot[applied={T|T < 7316333286377971712}]
I20260808 18:36:21.265367 11087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:21.265543 11087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:21.267879 11087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Sending DeleteTablet for 3 replicas of tablet 0549b4e3eb0b4540b1318c0cd4ec6d71
I20260808 18:36:21.268723 11352 tablet_service.cc:1514] Processing DeleteTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:55342
I20260808 18:36:21.268726 11216 tablet_service.cc:1514] Processing DeleteTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:60844
I20260808 18:36:21.268761 11493 tablet_service.cc:1514] Processing DeleteTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:45846
I20260808 18:36:21.269227 11606 tablet_replica.cc:330] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: stopping tablet replica
I20260808 18:36:21.269268 11607 tablet_replica.cc:330] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: stopping tablet replica
I20260808 18:36:21.269519 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11152
I20260808 18:36:21.269747 11607 raft_consensus.cc:2229] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:21.270147 11607 raft_consensus.cc:2258] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:21.273563 11608 tablet_replica.cc:330] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: stopping tablet replica
I20260808 18:36:21.273860 11608 raft_consensus.cc:2229] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:36:21.274080 11607 ts_tablet_manager.cc:1905] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:21.274231 11608 raft_consensus.cc:2258] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:21.275627 11608 ts_tablet_manager.cc:1905] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:21.277197 11607 ts_tablet_manager.cc:1918] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:21.277359 11607 log.cc:1196] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/0549b4e3eb0b4540b1318c0cd4ec6d71
I20260808 18:36:21.277774 11607 ts_tablet_manager.cc:1939] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P 5e60d8245560480c8c80044501e178fc: Deleting consensus metadata
I20260808 18:36:21.278525 11608 ts_tablet_manager.cc:1918] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:21.278645 11608 log.cc:1196] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/0549b4e3eb0b4540b1318c0cd4ec6d71
I20260808 18:36:21.278976 11608 ts_tablet_manager.cc:1939] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P e56f0e39bb224f5aaf0380719e86f4ff: Deleting consensus metadata
I20260808 18:36:21.279399 11075 catalog_manager.cc:4841] TS 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (table test-table [id=2aa254b349cf486b9013fecbd524212f]) successfully deleted
I20260808 18:36:21.280093 11074 catalog_manager.cc:4841] TS e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (table test-table [id=2aa254b349cf486b9013fecbd524212f]) successfully deleted
W20260808 18:36:21.281106 11075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:41825 (error 108)
I20260808 18:36:21.281394 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11288
I20260808 18:36:21.283319 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.282999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.0.0.1:60850)
0808 18:36:21.283131 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.283212 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 18:36:21.283496 11075 catalog_manager.cc:4584] TS c314e40244c647968079c60718c0c802 (127.27.251.193:41825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71: Network error: Client connection negotiation failed: client connection to 127.27.251.193:41825: connect: Connection refused (error 111)
I20260808 18:36:21.289728 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11426
I20260808 18:36:21.298563 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11056
I20260808 18:36:21.306509 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--webserver_interface=127.27.251.254
--webserver_port=33629
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:21.422137 11611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:21.422463 11611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:21.422544 11611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:21.426604 11611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:21.426700 11611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:21.426739 11611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:21.426776 11611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:21.426827 11611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:21.432065 11611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:21.433143 11611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:21.434216 11611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:21.450387 11617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:21.450484 11611 server_base.cc:1034] running on GCE node
W20260808 18:36:21.450385 11620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:21.450385 11618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:21.451357 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:21.452574 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:21.453769 11611 hybrid_clock.cc:648] HybridClock initialized: now 1786214181453736 us; error 44 us; skew 500 ppm
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:21.461088 11611 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:21.462072 11611 webserver.cc:460] Webserver started at http://127.27.251.254:33629/ using document root <none> and password file <none>
I20260808 18:36:21.462934 11611 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:21.463045 11611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:21.469429 11611 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:36:21.471863 11627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:21.473349 11611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:21.473528 11611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "ed97245b92d348bf8cd2184c724fe23d"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "1543185eb856b17ce4b25c72084785b9"
server_key_iv: "dd55de814a2cc0884b952f6118f994c2"
server_key_version: "encryptionkey@0"
I20260808 18:36:21.474213 11611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:21.485562 11611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:21.489293 11611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:21.489523 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:21.497296 11611 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:41795
I20260808 18:36:21.497615 11681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:41795 every 8 connection(s)
I20260808 18:36:21.498602 11611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:21.503175 11682 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:36:21.503793 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11611
I20260808 18:36:21.504311 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:41825
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=40831
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:21.505450 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap starting.
I20260808 18:36:21.523952 11682 log.cc:824] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:21.532971 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:36:21.533372 11682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap complete.
I20260808 18:36:21.537710 11682 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:21.538091 11682 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Initialized, Role: FOLLOWER
I20260808 18:36:21.538574 11682 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:21.538686 11682 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:21.538780 11682 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:21.538893 11682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:36:21.540910 11682 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:21.541276 11682 leader_election.cc:302] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed97245b92d348bf8cd2184c724fe23d; no voters:
I20260808 18:36:21.541638 11682 leader_election.cc:288] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:36:21.541724 11686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:36:21.541921 11686 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 LEADER]: Becoming Leader. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Running, Role: LEADER
I20260808 18:36:21.542239 11686 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:21.542923 11682 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:21.543195 11686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed97245b92d348bf8cd2184c724fe23d. Latest consensus state: current_term: 2 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:21.543310 11686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:21.543670 11690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:21.543795 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:21.543977 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:21.548463 11690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:21.549348 11690 catalog_manager.cc:1259] Loaded cluster ID: 44a2ea6d0f2c4e228996f8d8a3e512a1
I20260808 18:36:21.549412 11690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:21.550879 11690 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 18:36:21.555845 11690 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 18:36:21.561462 11690 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:21.562230 11690 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Loaded TSK: 0
I20260808 18:36:21.563400 11690 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:36:21.640789 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:21.641165 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:21.641247 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:21.646435 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:21.646507 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:21.646588 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:21.652460 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:41825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=40831
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:21.653609 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:21.654685 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:21.665601 11710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:21.665717 11711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:21.665633 11684 server_base.cc:1034] running on GCE node
W20260808 18:36:21.665886 11713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:21.666384 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:21.667135 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:21.668344 11684 hybrid_clock.cc:648] HybridClock initialized: now 1786214181668323 us; error 37 us; skew 500 ppm
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:21.671540 11684 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:21.672499 11684 webserver.cc:460] Webserver started at http://127.27.251.193:40831/ using document root <none> and password file <none>
I20260808 18:36:21.673156 11684 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:21.673269 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:21.677397 11684 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:21.679869 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:21.681267 11684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:21.681442 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c314e40244c647968079c60718c0c802"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "5ab0ab0578c313e26b225fbf78ba873f"
server_key_iv: "fe5c2514f2e3418403edf57376d30939"
server_key_version: "encryptionkey@0"
I20260808 18:36:21.682094 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:21.713209 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:21.717147 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:21.717381 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:21.718047 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:21.719209 11727 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:36:21.723359 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:36:21.723470 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260808 18:36:21.723598 11684 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:36:21.725340 11684 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:36:21.725411 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260808 18:36:21.725697 11727 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Bootstrap starting.
I20260808 18:36:21.735929 11684 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:41825
I20260808 18:36:21.736066 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:41825 every 8 connection(s)
I20260808 18:36:21.737087 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:21.742727 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11684
I20260808 18:36:21.743242 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:46513
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=44937
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:21.747172 11727 log.cc:824] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:21.756419 11727 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:36:21.756830 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.739170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:36429 (local address 127.27.251.254:41795)
0808 18:36:21.739572 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.739591 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.740281 (+ 690us) server_negotiation.cc:407] Connection header received
0808 18:36:21.741138 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.741164 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.741650 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.742059 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:21.743615 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.745299 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.746574 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.747052 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.747176 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.750526 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:21.750533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:21.750550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:21.750583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:21.752759 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.753494 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.753500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.753506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.753662 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.754134 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.754137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.754139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.754587 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:21.754729 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.755476 (+ 747us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.755777 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260808 18:36:21.756901 11727 tablet_bootstrap.cc:492] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Bootstrap complete.
I20260808 18:36:21.756908 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.739071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:36429)
0808 18:36:21.739732 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.739784 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.740673 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.742123 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.742129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.742658 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:21.743333 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.743357 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.745481 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.745484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.746371 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.746430 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.747202 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.747206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.747404 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.748113 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:21.748130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:21.750282 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:21.752899 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.752909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.752926 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.753353 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.753784 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.753789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.753793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.754009 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.754740 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:21.754750 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:21.755100 (+ 350us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.755344 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.755766 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":148,"threads_started":1}
I20260808 18:36:21.758457 11727 ts_tablet_manager.cc:1397] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.009s
I20260808 18:36:21.759703 11840 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:21.759938 11840 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:21.760476 11840 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
I20260808 18:36:21.761279 11727 raft_consensus.cc:348] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.761780 11727 raft_consensus.cc:729] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c314e40244c647968079c60718c0c802, State: Initialized, Role: FOLLOWER
I20260808 18:36:21.762310 11727 consensus_queue.cc:260] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } }
I20260808 18:36:21.763229 11727 ts_tablet_manager.cc:1428] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:36:21.763230 11645 ts_manager.cc:194] Registered new tserver with Master: c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:21.767637 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:36429
I20260808 18:36:21.770303 11840 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:36:21.777438 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.766572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.0.0.1:60860)
0808 18:36:21.766947 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.766988 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.767121 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.767791 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.767801 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.768149 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:21.768268 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.768280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.769749 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.769755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.770729 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.770741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.771372 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.771376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.771503 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.772075 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:21.772087 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:21.773736 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:21.775570 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.775579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.775585 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.775811 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.776133 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.776136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.776138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.776226 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.776778 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:21.776784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:21.776911 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.777056 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.777256 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":119,"threads_started":1}
I20260808 18:36:21.777457 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.766616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.27.251.193:41825)
0808 18:36:21.767317 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.767327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.767343 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:36:21.767395 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.767398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.767571 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.767722 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:21.768452 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.769546 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.770847 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.771237 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.771368 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.773876 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:21.773886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:21.773899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:21.773929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:21.775452 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.775932 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.775938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.775940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.776027 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.776322 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.776326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.776328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.776657 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:21.776805 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.777144 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.777314 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":99,"threads_started":1}
I20260808 18:36:21.778451 11772 tablet_service.cc:1514] Processing DeleteTablet for tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:21 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:60860
I20260808 18:36:21.779448 11849 tablet_replica.cc:330] stopping tablet replica
I20260808 18:36:21.779776 11849 raft_consensus.cc:2229] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:21.780036 11849 raft_consensus.cc:2258] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:21.781080 11849 ts_tablet_manager.cc:1905] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:21.784760 11849 ts_tablet_manager.cc:1918] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:21.784880 11849 log.cc:1196] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/0549b4e3eb0b4540b1318c0cd4ec6d71
I20260808 18:36:21.785195 11849 ts_tablet_manager.cc:1939] T 0549b4e3eb0b4540b1318c0cd4ec6d71 P c314e40244c647968079c60718c0c802: Deleting consensus metadata
I20260808 18:36:21.786955 11632 catalog_manager.cc:4841] TS c314e40244c647968079c60718c0c802 (127.27.251.193:41825): tablet 0549b4e3eb0b4540b1318c0cd4ec6d71 (table test-table [id=2aa254b349cf486b9013fecbd524212f]) successfully deleted
W20260808 18:36:21.869552 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:21.869807 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:21.869850 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:21.874017 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:21.874078 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:21.874136 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:21.879642 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:46513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=44937
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:21.880760 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:21.881862 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:21.891914 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:21.891937 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:21.891961 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:21.892010 11844 server_base.cc:1034] running on GCE node
I20260808 18:36:21.892485 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:21.893211 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:21.894399 11844 hybrid_clock.cc:648] HybridClock initialized: now 1786214181894369 us; error 47 us; skew 500 ppm
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:21.897186 11844 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:21.898082 11844 webserver.cc:460] Webserver started at http://127.27.251.194:44937/ using document root <none> and password file <none>
I20260808 18:36:21.898655 11844 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:21.898839 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:21.902719 11844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:21.904834 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:21.906139 11844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:36:21.906286 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "9141e7d544a7c72950cd31323b21a9de"
server_key_iv: "a36b05f1fe0f0b59e5287ae9ee69ebc2"
server_key_version: "encryptionkey@0"
I20260808 18:36:21.906875 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:21.937218 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:21.940299 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:21.940567 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:21.941205 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:21.942066 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:21.942137 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:21.942222 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:21.942274 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:21.951529 11844 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:46513
I20260808 18:36:21.951556 11981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:46513 every 8 connection(s)
I20260808 18:36:21.952638 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
Aug 08 18:36:21 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:21.962875 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11844
I20260808 18:36:21.963407 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:46373
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=44385
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:21.967597 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.954570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:44939 (local address 127.27.251.254:41795)
0808 18:36:21.954729 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:21.954733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:21.955604 (+ 871us) server_negotiation.cc:407] Connection header received
0808 18:36:21.957081 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:21.957085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:21.957147 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:21.957260 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:21.958492 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.959039 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.960219 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.960464 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.960628 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.963246 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:21.963251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:21.963253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:21.963287 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:21.965218 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.965890 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.965895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.965898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.966045 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:21.966389 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:21.966394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:21.966397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:21.966665 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:21.966776 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:21.967267 (+ 491us) server_negotiation.cc:299] Negotiation successful
0808 18:36:21.967409 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:36:21.968502 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:21.954997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:44939)
0808 18:36:21.955439 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:21.955483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:21.956847 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:21.957454 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:21.957461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:21.957843 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:21.958333 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.958347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.959173 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.959179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.960077 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:21.960087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:21.960684 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:21.960687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:21.960865 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:21.961498 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:21.961516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:21.963068 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:21.965363 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.965372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.965390 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.965772 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.966149 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:21.966152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:21.966153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:21.966292 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:21.966771 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:21.966776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:21.967009 (+ 233us) client_negotiation.cc:769] Sending connection context
0808 18:36:21.967213 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 18:36:21.967544 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260808 18:36:21.969755 11982 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:21.970013 11982 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:21.970520 11982 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
I20260808 18:36:21.971614 11645 ts_manager.cc:194] Registered new tserver with Master: e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
I20260808 18:36:21.972186 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:44939
W20260808 18:36:22.081629 11985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:22.081907 11985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:22.081997 11985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:22.086081 11985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:22.086169 11985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:22.086259 11985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:22.091696 11985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=44385
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:22.092842 11985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:22.093909 11985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:22.103844 11991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:22.103856 11990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:22.103945 11993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:22.104184 11985 server_base.cc:1034] running on GCE node
I20260808 18:36:22.104684 11985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:22.105362 11985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:22.106588 11985 hybrid_clock.cc:648] HybridClock initialized: now 1786214182106572 us; error 65 us; skew 500 ppm
Aug 08 18:36:22 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:22.109673 11985 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:22.110632 11985 webserver.cc:460] Webserver started at http://127.27.251.195:44385/ using document root <none> and password file <none>
I20260808 18:36:22.111266 11985 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:22.111348 11985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:22.115271 11985 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:22.117519 12000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:22.118772 11985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:22.118906 11985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "5e60d8245560480c8c80044501e178fc"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "406cacffb74639e2f8809250c13c5515"
server_key_iv: "1726bbd16811e38896deace1541d5292"
server_key_version: "encryptionkey@0"
I20260808 18:36:22.119495 11985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:22.150661 11985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:22.153896 11985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:22.154137 11985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:22.154829 11985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:22.155766 11985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:22.155818 11985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:22.155896 11985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:22.155936 11985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:22.166775 11985 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:46373
I20260808 18:36:22.166796 12118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:46373 every 8 connection(s)
I20260808 18:36:22.167883 11985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:22.172094 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 11985
Aug 08 18:36:22 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:22.181679 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.172850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.27.251.254:41795)
0808 18:36:22.173103 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.173108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.173293 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:36:22.173461 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.173464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.173506 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.173626 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:22.174359 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.174870 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.175743 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.176035 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.176163 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.177650 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:22.177654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:22.177657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:22.177682 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:22.179700 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:22.180197 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:22.180203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:22.180207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:22.180316 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:22.180669 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:22.180674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:22.180676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:22.180982 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:22.181229 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.181316 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.181452 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260808 18:36:22.183519 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.169849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:45655 (local address 127.27.251.254:41795)
0808 18:36:22.170023 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.170028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.170868 (+ 840us) server_negotiation.cc:407] Connection header received
0808 18:36:22.172275 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.172278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.172337 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.172431 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:22.173939 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.174485 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.175697 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.175993 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.176119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.179142 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:22.179152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:22.179154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:22.179178 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:22.181039 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:22.181673 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:22.181675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:22.181676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:22.181772 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:22.182265 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:22.182270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:22.182273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:22.182486 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:22.182613 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.183214 (+ 601us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.183340 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":30464}
I20260808 18:36:22.184330 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.170228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:45655)
0808 18:36:22.170709 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.170754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:22.172017 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:22.172720 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:22.172730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:22.173244 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:22.173757 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.173772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.174629 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.174633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.175552 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.175563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.176161 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.176167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.176365 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.177046 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:22.177064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:22.178961 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:22.181174 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:22.181180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:22.181196 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:22.181566 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:22.181911 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:22.181915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:22.181918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:22.182111 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:22.182630 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:22.182636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:22.182966 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 18:36:22.183130 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 18:36:22.183384 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260808 18:36:22.185459 12119 heartbeater.cc:347] Connected to a master server at 127.27.251.254:41795
I20260808 18:36:22.185688 12119 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:22.186218 12119 heartbeater.cc:511] Master 127.27.251.254:41795 requested a full tablet report, sending...
I20260808 18:36:22.187227 11645 ts_manager.cc:194] Registered new tserver with Master: 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373)
I20260808 18:36:22.187745 11645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:45655
I20260808 18:36:22.195776 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:36:22.200132 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.196441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.27.251.254:41795)
0808 18:36:22.196616 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.196620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.196810 (+ 190us) server_negotiation.cc:407] Connection header received
0808 18:36:22.196932 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.196934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.197038 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.197123 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:36:22.197395 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.197911 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.198847 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.199080 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.199161 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.199423 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:36:22.199802 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:36:22.199893 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.199969 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.200016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 18:36:22.200963 11645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:36:22.204356 11645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:36:22.212939 11772 tablet_service.cc:1467] Processing CreateTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b (DEFAULT_TABLE table=test-table [id=1fde0374cc74428daa215a2597f9a9b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:22.213630 11772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865a0ffe9a2483aaca90026ecd2004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:22.216279 11727 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Bootstrap starting.
I20260808 18:36:22.217208 11727 tablet_bootstrap.cc:654] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:22.219133 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.212315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:46513 (local address 127.0.0.1:55358)
0808 18:36:22.212623 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.212637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:22.212728 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:22.213619 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:22.213625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:22.213650 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:22.213862 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.213869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.215641 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.215644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.217488 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.217503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.218652 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.218656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.218764 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.218805 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 18:36:22.218855 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 18:36:22.218973 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":190,"thread_start_us":150,"threads_started":1}
I20260808 18:36:22.219187 11727 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: No bootstrap required, opened a new log
I20260808 18:36:22.219266 11727 ts_tablet_manager.cc:1397] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:22.219373 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.212322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.27.251.195:46373)
0808 18:36:22.213171 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.213182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.213207 (+ 25us) server_negotiation.cc:407] Connection header received
0808 18:36:22.213287 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.213292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.213522 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.213905 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:22.214063 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.215484 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.217580 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.218365 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.218505 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.218665 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.219066 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.219186 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":109,"threads_started":1}
I20260808 18:36:22.219445 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.211883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46373 (local address 127.0.0.1:45856)
0808 18:36:22.212018 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.212040 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:22.212137 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:22.213712 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:22.213718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:22.213733 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:22.213900 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.213913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.215641 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.215644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.217372 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.217387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.218520 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.218524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.218654 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.218740 (+ 86us) client_negotiation.cc:769] Sending connection context
0808 18:36:22.218819 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 18:36:22.218935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":328}
I20260808 18:36:22.219759 11727 raft_consensus.cc:348] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.219882 11727 raft_consensus.cc:374] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:22.219892 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.212708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.27.251.194:46513)
0808 18:36:22.213098 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.213114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.213140 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:36:22.213220 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.213226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.213443 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.213612 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:22.214054 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.215445 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.217639 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.218506 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.218956 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.219115 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.219163 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.219775 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260808 18:36:22.219936 11727 raft_consensus.cc:729] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c314e40244c647968079c60718c0c802, State: Initialized, Role: FOLLOWER
I20260808 18:36:22.220903 11727 consensus_queue.cc:260] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.221649 11727 ts_tablet_manager.cc:1428] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 18:36:22.222940 12050 tablet_service.cc:1467] Processing CreateTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b (DEFAULT_TABLE table=test-table [id=1fde0374cc74428daa215a2597f9a9b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:22.222940 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b (DEFAULT_TABLE table=test-table [id=1fde0374cc74428daa215a2597f9a9b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:22.224110 12050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865a0ffe9a2483aaca90026ecd2004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:22.224126 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1865a0ffe9a2483aaca90026ecd2004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:22.230451 12133 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Bootstrap starting.
I20260808 18:36:22.230616 12134 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Bootstrap starting.
I20260808 18:36:22.233201 12133 tablet_bootstrap.cc:654] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:22.233347 12134 tablet_bootstrap.cc:654] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:22.233971 12133 log.cc:824] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:22.234112 12134 log.cc:824] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:22.236078 12134 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: No bootstrap required, opened a new log
I20260808 18:36:22.236150 12133 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: No bootstrap required, opened a new log
I20260808 18:36:22.236275 12134 ts_tablet_manager.cc:1397] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:36:22.236369 12133 ts_tablet_manager.cc:1397] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260808 18:36:22.238292 11841 tablet.cc:2369] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:22.239315 12134 raft_consensus.cc:348] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.239564 12134 raft_consensus.cc:374] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:22.239656 12134 raft_consensus.cc:729] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56f0e39bb224f5aaf0380719e86f4ff, State: Initialized, Role: FOLLOWER
I20260808 18:36:22.239861 12133 raft_consensus.cc:348] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.240154 12134 consensus_queue.cc:260] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.240250 12133 raft_consensus.cc:374] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:22.240319 12133 raft_consensus.cc:729] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e60d8245560480c8c80044501e178fc, State: Initialized, Role: FOLLOWER
I20260808 18:36:22.240837 12133 consensus_queue.cc:260] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.240976 11982 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
I20260808 18:36:22.241268 12134 ts_tablet_manager.cc:1428] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:36:22.241583 12119 heartbeater.cc:503] Master 127.27.251.254:41795 was elected leader, sending a full tablet report...
I20260808 18:36:22.241902 12133 ts_tablet_manager.cc:1428] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260808 18:36:22.419019 12120 tablet.cc:2369] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:22.453691 12138 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 18:36:22.453882 11983 tablet.cc:2369] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:36:22.453939 12138 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.455299 12138 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513), c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:22.460284 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.455706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:36835 (local address 127.27.251.193:41825)
0808 18:36:22.455985 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.455989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.456132 (+ 143us) server_negotiation.cc:407] Connection header received
0808 18:36:22.456288 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.456297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.456354 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.456470 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:22.456991 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.457709 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.459072 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.459722 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.459839 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.459969 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.460080 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.460150 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":69,"threads_started":1}
I20260808 18:36:22.460282 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.455610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:36239 (local address 127.27.251.194:46513)
0808 18:36:22.455742 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.455745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.456023 (+ 278us) server_negotiation.cc:407] Connection header received
0808 18:36:22.456193 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.456203 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.456248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.456333 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:22.456636 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.457452 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.458985 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.459599 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.459726 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.459799 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.460071 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.460152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:36:22.460299 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.455550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:46513 (local address 127.27.251.195:36239)
0808 18:36:22.455930 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.455945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:22.456073 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:22.456342 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:22.456344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:22.456361 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:22.456477 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.456484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.457575 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.457578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.458849 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.458860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.459741 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.459744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.459898 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.459928 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 18:36:22.460020 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:36:22.460108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291}
I20260808 18:36:22.460306 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.455598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.27.251.195:36835)
0808 18:36:22.456023 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.456037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:22.456175 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:22.456720 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:22.456722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:22.456737 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:22.456858 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.456865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.457823 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.457826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.458946 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:22.458954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.459845 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.459850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:22.459953 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.459986 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 18:36:22.460063 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 18:36:22.460109 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
I20260808 18:36:22.461009 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865a0ffe9a2483aaca90026ecd2004b" candidate_uuid: "5e60d8245560480c8c80044501e178fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" is_pre_election: true
I20260808 18:36:22.461182 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865a0ffe9a2483aaca90026ecd2004b" candidate_uuid: "5e60d8245560480c8c80044501e178fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c314e40244c647968079c60718c0c802" is_pre_election: true
I20260808 18:36:22.461360 11936 raft_consensus.cc:2454] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e60d8245560480c8c80044501e178fc in term 0.
I20260808 18:36:22.461572 11792 raft_consensus.cc:2454] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e60d8245560480c8c80044501e178fc in term 0.
I20260808 18:36:22.461938 12003 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff; no voters:
I20260808 18:36:22.462266 12138 raft_consensus.cc:2781] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:36:22.462385 12138 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:36:22.462441 12138 raft_consensus.cc:3037] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:22.463634 12138 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.464030 12138 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [CANDIDATE]: Term 1 election: Requested vote from peers e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513), c314e40244c647968079c60718c0c802 (127.27.251.193:41825)
I20260808 18:36:22.464397 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865a0ffe9a2483aaca90026ecd2004b" candidate_uuid: "5e60d8245560480c8c80044501e178fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c314e40244c647968079c60718c0c802"
I20260808 18:36:22.464416 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1865a0ffe9a2483aaca90026ecd2004b" candidate_uuid: "5e60d8245560480c8c80044501e178fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
I20260808 18:36:22.464602 11936 raft_consensus.cc:3037] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:22.464597 11792 raft_consensus.cc:3037] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:22.465654 11792 raft_consensus.cc:2454] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e60d8245560480c8c80044501e178fc in term 1.
I20260808 18:36:22.465672 11936 raft_consensus.cc:2454] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e60d8245560480c8c80044501e178fc in term 1.
I20260808 18:36:22.465952 12003 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff; no voters:
I20260808 18:36:22.466148 12138 raft_consensus.cc:2781] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:22.466361 12138 raft_consensus.cc:686] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 1 LEADER]: Becoming Leader. State: Replica: 5e60d8245560480c8c80044501e178fc, State: Running, Role: LEADER
I20260808 18:36:22.466732 12138 consensus_queue.cc:237] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:22.470094 11645 catalog_manager.cc:5495] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e60d8245560480c8c80044501e178fc (127.27.251.195). New cstate: current_term: 1 leader_uuid: "5e60d8245560480c8c80044501e178fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } health_report { overall_health: UNKNOWN } } }
I20260808 18:36:22.523418 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.519821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.27.251.195:46373)
0808 18:36:22.519961 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:22.519964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:22.519996 (+ 32us) server_negotiation.cc:407] Connection header received
0808 18:36:22.520081 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:22.520084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:22.520145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:22.520237 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:36:22.520496 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.521027 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.521853 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:22.522068 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:22.522168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:22.522445 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:36:22.522936 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:36:22.523078 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:22.523151 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 18:36:22.523207 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 18:36:22.529891 11792 raft_consensus.cc:1261] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Refusing update from remote peer 5e60d8245560480c8c80044501e178fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:22.529896 11936 raft_consensus.cc:1261] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Refusing update from remote peer 5e60d8245560480c8c80044501e178fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:22.530758 12138 consensus_queue.cc:1035] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:22.531035 12141 consensus_queue.cc:1035] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:22.539737 12145 mvcc.cc:205] Tried to move back new op lower bound from 7316333291633582080 to 7316333291386773504. Current Snapshot: MvccSnapshot[applied={T|T < 7316333291633582080 or (T in {7316333291633582080})}]
I20260808 18:36:22.546772 11643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:22.546929 11643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:22.549554 11643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Sending DeleteTablet for 3 replicas of tablet 1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:22.550387 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:22 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:55358
I20260808 18:36:22.550514 11772 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:22 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:60860
I20260808 18:36:22.550607 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11684
I20260808 18:36:22.550933 12050 tablet_service.cc:1514] Processing DeleteTablet for tablet 1865a0ffe9a2483aaca90026ecd2004b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:22 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:45856
I20260808 18:36:22.552817 12154 tablet_replica.cc:330] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: stopping tablet replica
I20260808 18:36:22.553167 12154 raft_consensus.cc:2229] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:22.553568 12154 raft_consensus.cc:2258] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:22.554241 12155 tablet_replica.cc:330] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: stopping tablet replica
I20260808 18:36:22.554633 12155 raft_consensus.cc:2229] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:36:22.555151 12155 raft_consensus.cc:2258] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:22.557543 12154 ts_tablet_manager.cc:1905] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:22.559813 12154 ts_tablet_manager.cc:1918] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:22.559885 12154 log.cc:1196] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:22.560132 12154 ts_tablet_manager.cc:1939] T 1865a0ffe9a2483aaca90026ecd2004b P e56f0e39bb224f5aaf0380719e86f4ff: Deleting consensus metadata
I20260808 18:36:22.561175 12155 ts_tablet_manager.cc:1905] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:22.561707 11631 catalog_manager.cc:4841] TS e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): tablet 1865a0ffe9a2483aaca90026ecd2004b (table test-table [id=1fde0374cc74428daa215a2597f9a9b6]) successfully deleted
I20260808 18:36:22.563577 12155 ts_tablet_manager.cc:1918] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:22.563699 12155 log.cc:1196] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:22.564013 12155 ts_tablet_manager.cc:1939] T 1865a0ffe9a2483aaca90026ecd2004b P 5e60d8245560480c8c80044501e178fc: Deleting consensus metadata
I20260808 18:36:22.565172 11632 catalog_manager.cc:4841] TS 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): tablet 1865a0ffe9a2483aaca90026ecd2004b (table test-table [id=1fde0374cc74428daa215a2597f9a9b6]) successfully deleted
W20260808 18:36:22.565748 11632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:41825 (error 108)
I20260808 18:36:22.566342 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11844
I20260808 18:36:22.567855 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:22.566536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:41825 (local address 127.0.0.1:60876)
0808 18:36:22.567682 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:22.567766 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:41825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1035}
W20260808 18:36:22.568249 11632 catalog_manager.cc:4584] TS c314e40244c647968079c60718c0c802 (127.27.251.193:41825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1865a0ffe9a2483aaca90026ecd2004b: Network error: Client connection negotiation failed: client connection to 127.27.251.193:41825: connect: Connection refused (error 111)
I20260808 18:36:22.574308 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11985
I20260808 18:36:22.582417 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 11611
I20260808 18:36:22.591476 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--webserver_interface=127.27.251.254
--webserver_port=33629
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:22.710453 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:22.710722 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:22.710762 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:22.714773 12157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:22.714826 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:22.714845 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:22.714861 12157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:22.714876 12157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:22.720309 12157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:22.721333 12157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:22.722312 12157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:22.730919 12162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:22.730922 12165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:22.731125 12157 server_base.cc:1034] running on GCE node
W20260808 18:36:22.730891 12163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:22.731835 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:22.733084 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:22.734256 12157 hybrid_clock.cc:648] HybridClock initialized: now 1786214182734226 us; error 40 us; skew 500 ppm
Aug 08 18:36:22 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:22.737185 12157 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:22.738083 12157 webserver.cc:460] Webserver started at http://127.27.251.254:33629/ using document root <none> and password file <none>
I20260808 18:36:22.738682 12157 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:22.738744 12157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:22.742553 12157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:22.744661 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:22.745911 12157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:22.746078 12157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "ed97245b92d348bf8cd2184c724fe23d"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "1543185eb856b17ce4b25c72084785b9"
server_key_iv: "dd55de814a2cc0884b952f6118f994c2"
server_key_version: "encryptionkey@0"
I20260808 18:36:22.746613 12157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:22.756388 12157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:22.759292 12157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:22.759503 12157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:22.766908 12157 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:41795
I20260808 18:36:22.767295 12228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:41795 every 8 connection(s)
I20260808 18:36:22.768047 12157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:22.768565 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12157
I20260808 18:36:22.768994 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:41825
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=40831
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:22.772487 12229 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:36:22.774210 12229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap starting.
I20260808 18:36:22.792860 12229 log.cc:824] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:22.804199 12229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:36:22.804651 12229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d: Bootstrap complete.
I20260808 18:36:22.808964 12229 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:22.809374 12229 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Initialized, Role: FOLLOWER
I20260808 18:36:22.809911 12229 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:22.810074 12229 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:22.810170 12229 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:22.810268 12229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:36:22.812136 12229 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:22.812474 12229 leader_election.cc:302] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed97245b92d348bf8cd2184c724fe23d; no voters:
I20260808 18:36:22.812889 12229 leader_election.cc:288] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 18:36:22.812976 12233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 3 FOLLOWER]: Leader election won for term 3
I20260808 18:36:22.813191 12233 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [term 3 LEADER]: Becoming Leader. State: Replica: ed97245b92d348bf8cd2184c724fe23d, State: Running, Role: LEADER
I20260808 18:36:22.813482 12233 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } }
I20260808 18:36:22.814152 12229 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:22.814855 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:22.814985 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:22.815927 12235 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed97245b92d348bf8cd2184c724fe23d. Latest consensus state: current_term: 3 leader_uuid: "ed97245b92d348bf8cd2184c724fe23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed97245b92d348bf8cd2184c724fe23d" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 41795 } } }
I20260808 18:36:22.816051 12235 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed97245b92d348bf8cd2184c724fe23d [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:22.815982 12241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:22.821333 12241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:22.822041 12241 catalog_manager.cc:1259] Loaded cluster ID: 44a2ea6d0f2c4e228996f8d8a3e512a1
I20260808 18:36:22.822118 12241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 18:36:22.823024 12241 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fef991369ad google::LogMessage::Fail() at ??:0
@ 0x7fef9913988a google::LogMessage::SendToLog() at ??:0
@ 0x7fef991364de google::LogMessage::Flush() at ??:0
@ 0x7fef99137a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fefa3a75fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fefa3a76d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fefa3a73579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fefa3ab7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fef9b3883d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fef9b388c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fef9b38a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fef9b379512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fef9c9bd6db start_thread at ??:0
@ 0x7fef97c3461f clone at ??:0
W20260808 18:36:22.920547 12231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:22.920845 12231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:22.920917 12231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:22.925125 12231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:22.925182 12231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:22.925266 12231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:22.931062 12231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:41825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=40831
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:22.932338 12231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:22.933537 12231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:22.943848 12270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:22.943858 12271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:22.944152 12273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:22.944681 12231 server_base.cc:1034] running on GCE node
I20260808 18:36:22.945125 12231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:22.945863 12231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:22.947082 12231 hybrid_clock.cc:648] HybridClock initialized: now 1786214182947062 us; error 38 us; skew 500 ppm
Aug 08 18:36:22 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:22.950100 12231 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:22.951085 12231 webserver.cc:460] Webserver started at http://127.27.251.193:40831/ using document root <none> and password file <none>
I20260808 18:36:22.951715 12231 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:22.951794 12231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:22.955951 12231 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:22.958387 12282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:22.959898 12231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:36:22.960075 12231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "c314e40244c647968079c60718c0c802"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "5ab0ab0578c313e26b225fbf78ba873f"
server_key_iv: "fe5c2514f2e3418403edf57376d30939"
server_key_version: "encryptionkey@0"
I20260808 18:36:22.960788 12231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:23.005156 12231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:23.009197 12231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:23.009461 12231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:23.010210 12231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:23.011475 12289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:36:23.015797 12231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:36:23.015897 12231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260808 18:36:23.015992 12231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:36:23.018079 12231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:36:23.018157 12231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260808 18:36:23.018471 12289 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Bootstrap starting.
I20260808 18:36:23.030570 12231 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:41825
I20260808 18:36:23.030622 12400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:41825 every 8 connection(s)
I20260808 18:36:23.031670 12231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:23.038148 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12231
I20260808 18:36:23.038655 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:46513
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=44937
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:36:23.041456 12289 log.cc:824] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:23.049814 12289 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:36:23.050267 12289 tablet_bootstrap.cc:492] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Bootstrap complete.
I20260808 18:36:23.051628 12289 ts_tablet_manager.cc:1397] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260808 18:36:23.055956 12289 raft_consensus.cc:348] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:23.056752 12289 raft_consensus.cc:729] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c314e40244c647968079c60718c0c802, State: Initialized, Role: FOLLOWER
I20260808 18:36:23.057384 12289 consensus_queue.cc:260] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:23.058723 12289 ts_tablet_manager.cc:1428] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
W20260808 18:36:23.062673 12404 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:36:23.033903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:54727)
0808 18:36:23.034444 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.034505 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:23.035430 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:23.061766 (+ 26336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":170,"threads_started":1}
W20260808 18:36:23.063279 12401 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:41795 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:41795: Client connection negotiation failed: client connection to 127.27.251.254:41795: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:36:23.064116 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.063832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:41451)
0808 18:36:23.063985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.064058 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 18:36:23.064991 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.064751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:48537)
0808 18:36:23.064894 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.064941 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 18:36:23.065235 12401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:36:23.163291 12405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:23.163564 12405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:23.163609 12405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:23.167889 12405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:23.167965 12405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:23.168056 12405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:23.173645 12405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:46513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=44937
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:23.174787 12405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:23.175904 12405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:23.185866 12412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:23.185874 12413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:23.185848 12415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:23.186128 12405 server_base.cc:1034] running on GCE node
I20260808 18:36:23.186682 12405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:23.187443 12405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:23.188608 12405 hybrid_clock.cc:648] HybridClock initialized: now 1786214183188571 us; error 47 us; skew 500 ppm
Aug 08 18:36:23 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:23.191440 12405 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:23.192351 12405 webserver.cc:460] Webserver started at http://127.27.251.194:44937/ using document root <none> and password file <none>
I20260808 18:36:23.192979 12405 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:23.193045 12405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:23.197008 12405 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:36:23.199270 12425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.200860 12405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:23.200964 12405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "e56f0e39bb224f5aaf0380719e86f4ff"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "9141e7d544a7c72950cd31323b21a9de"
server_key_iv: "a36b05f1fe0f0b59e5287ae9ee69ebc2"
server_key_version: "encryptionkey@0"
I20260808 18:36:23.201520 12405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:23.226281 12405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:23.229754 12405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:23.229929 12405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:23.230543 12405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:23.231365 12405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:23.231411 12405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.231453 12405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:23.231487 12405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.241196 12405 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:46513
I20260808 18:36:23.241217 12546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:46513 every 8 connection(s)
I20260808 18:36:23.242272 12405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:36:23.246588 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.244431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:35961)
0808 18:36:23.245033 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.245156 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":117,"threads_started":1}
W20260808 18:36:23.247210 12547 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:41795 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:41795: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
I20260808 18:36:23.247869 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.247635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:47627)
0808 18:36:23.247760 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.247812 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 18:36:23.248297 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12405
I20260808 18:36:23.248889 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.248424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:48133)
0808 18:36:23.248730 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.248793 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
I20260808 18:36:23.248880 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:46373
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:41795
--webserver_port=44385
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:23.249069 12547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:36:23.367709 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:23.367992 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:23.368067 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:23.372043 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:23.372150 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:23.372239 12551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:23.377534 12551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=44385
--tserver_master_addrs=127.27.251.254:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:23.378623 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:23.379652 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:23.389443 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:23.389451 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:23.389451 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:23.389789 12551 server_base.cc:1034] running on GCE node
I20260808 18:36:23.390241 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:23.390895 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:23.392082 12551 hybrid_clock.cc:648] HybridClock initialized: now 1786214183392056 us; error 50 us; skew 500 ppm
Aug 08 18:36:23 dist-test-slave-fd7l krb5kdc[11042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:23.394860 12551 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:23.395774 12551 webserver.cc:460] Webserver started at http://127.27.251.195:44385/ using document root <none> and password file <none>
I20260808 18:36:23.396387 12551 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:23.396451 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:23.400230 12551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:36:23.402376 12566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.403637 12551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:23.403764 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "5e60d8245560480c8c80044501e178fc"
format_stamp: "Formatted at 2026-08-08 18:36:20 on dist-test-slave-fd7l"
server_key: "406cacffb74639e2f8809250c13c5515"
server_key_iv: "1726bbd16811e38896deace1541d5292"
server_key_version: "encryptionkey@0"
I20260808 18:36:23.404315 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:23.432438 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:23.435554 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:23.435779 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:23.436410 12551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:23.437307 12551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:23.437354 12551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.437394 12551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:23.437414 12551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:23.446557 12551 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:46373
I20260808 18:36:23.446581 12685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:46373 every 8 connection(s)
I20260808 18:36:23.447597 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:23.451705 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.449884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:35483)
0808 18:36:23.450359 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.450476 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
W20260808 18:36:23.452281 12686 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:41795 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:41795: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
I20260808 18:36:23.452967 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.452724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:33661)
0808 18:36:23.452850 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.452910 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 18:36:23.453588 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:23.453384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:51115)
0808 18:36:23.453494 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:23.453543 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:36:23.453776 12686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:36:23.457023 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12551
I20260808 18:36:24.066367 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.065798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:57605)
0808 18:36:24.066157 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:24.066247 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260808 18:36:24.250125 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.249707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:33817)
0808 18:36:24.249963 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:24.250021 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260808 18:36:24.454728 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.454331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:55133)
0808 18:36:24.454578 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:24.454633 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260808 18:36:24.553985 12693 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:24.554191 12693 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:24.555227 12693 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
Aug 08 18:36:24 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.194@KRBTEST.COM
Aug 08 18:36:24 dist-test-slave-fd7l krb5kdc[11042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.195@KRBTEST.COM
I20260808 18:36:24.569190 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.555536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:46513 (local address 127.27.251.193:40619)
0808 18:36:24.555645 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:24.555660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:24.555791 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:24.557693 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:24.557699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:24.558510 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:24.558693 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:24.558710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.559981 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.559985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:24.560979 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:24.560990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.561690 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:24.561951 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:24.562696 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:24.562713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:24.564386 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:24.566786 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:24.566792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:24.566806 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:24.567066 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:24.567413 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:24.567418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:24.567564 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:24.568198 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:24.568204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:24.568439 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 18:36:24.568678 (+ 239us) client_negotiation.cc:240] Negotiation successful
0808 18:36:24.569003 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":185}
I20260808 18:36:24.569226 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.555703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:40619 (local address 127.27.251.194:46513)
0808 18:36:24.556071 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:24.556092 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:24.556146 (+ 54us) server_negotiation.cc:407] Connection header received
0808 18:36:24.557012 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:24.557019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:24.557326 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:24.557643 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:24.558869 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.559841 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561109 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.561562 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561737 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:24.564732 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:24.564738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:24.564750 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:24.564778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:24.566661 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567192 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:24.567198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:24.567203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:24.567312 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567664 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:24.567669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:24.567671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:24.568071 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:24.568216 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:24.568796 (+ 580us) server_negotiation.cc:299] Negotiation successful
0808 18:36:24.569066 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260808 18:36:24.569705 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.555849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:49431 (local address 127.27.251.195:46373)
0808 18:36:24.556151 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:24.556178 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:24.556214 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:36:24.556955 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:24.556963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:24.557326 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:24.557649 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:24.558875 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.559834 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561102 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.561567 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561746 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:24.565140 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:24.565145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:24.565157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:24.565186 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:24.567129 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567601 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:24.567609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:24.567618 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:24.567728 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:24.568018 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:24.568022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:24.568024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:24.568405 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:24.568563 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:24.568997 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 18:36:24.569503 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":123,"threads_started":1}
I20260808 18:36:24.570067 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:24.555536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46373 (local address 127.27.251.193:49431)
0808 18:36:24.556019 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:24.556032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:24.556174 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:24.557695 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:24.557700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:24.558206 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:24.558695 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:24.558710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.559982 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.559986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:24.560975 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:24.560986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:24.561691 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:24.561694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:24.561949 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:24.562742 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:24.562754 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:24.564950 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:24.567251 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:24.567259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:24.567469 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:24.567830 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:24.567833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:24.567835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:24.567924 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:24.568535 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:24.568546 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:24.568662 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 18:36:24.568765 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 18:36:24.569906 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1200,"spinlock_wait_cycles":3840,"thread_start_us":65,"threads_started":1}
W20260808 18:36:24.572063 12285 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
W20260808 18:36:24.572227 12286 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:24.572320 12286 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c314e40244c647968079c60718c0c802; no voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff
I20260808 18:36:24.572667 12693 raft_consensus.cc:2726] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:36:25.067447 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:25.067165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:46163)
0808 18:36:25.067300 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:25.067360 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:36:25.251633 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:25.251166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:49681)
0808 18:36:25.251464 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:25.251527 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260808 18:36:25.455892 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:25.455451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:56801)
0808 18:36:25.455744 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:25.455801 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260808 18:36:26.068956 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:26.068469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:48617)
0808 18:36:26.068791 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:26.068849 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":147,"threads_started":1}
I20260808 18:36:26.253010 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:26.252481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:52141)
0808 18:36:26.252840 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:26.252911 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":87,"threads_started":1}
I20260808 18:36:26.457283 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:26.456787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:41131)
0808 18:36:26.457099 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:26.457164 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260808 18:36:26.714865 12702 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:26.715022 12702 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:26.715588 12702 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
W20260808 18:36:26.716584 12286 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
W20260808 18:36:26.716717 12285 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:26.716827 12285 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c314e40244c647968079c60718c0c802; no voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff
I20260808 18:36:26.716977 12702 raft_consensus.cc:2726] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:36:27.070277 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:27.069805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:57987)
0808 18:36:27.070094 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:27.070176 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260808 18:36:27.254263 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:27.253810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:56261)
0808 18:36:27.254111 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:27.254168 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260808 18:36:27.458585 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:27.458132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:35247)
0808 18:36:27.458430 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:27.458488 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":130,"threads_started":1}
I20260808 18:36:28.071856 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:28.071310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:56033)
0808 18:36:28.071651 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:28.071707 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":81,"threads_started":1}
I20260808 18:36:28.255607 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:28.255172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:50901)
0808 18:36:28.255440 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:28.255506 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260808 18:36:28.459928 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:28.459417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:44633)
0808 18:36:28.459734 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:28.459803 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260808 18:36:28.533776 12709 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:28.533918 12709 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:28.534341 12709 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
W20260808 18:36:28.535327 12285 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
W20260808 18:36:28.535454 12286 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:28.535527 12286 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c314e40244c647968079c60718c0c802; no voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff
I20260808 18:36:28.535681 12709 raft_consensus.cc:2726] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:36:29.073298 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:29.072744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:45567)
0808 18:36:29.073060 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:29.073153 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260808 18:36:29.257051 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:29.256620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:57889)
0808 18:36:29.256885 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:29.256946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260808 18:36:29.461162 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:29.460737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:50911)
0808 18:36:29.461003 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:29.461065 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260808 18:36:30.074518 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:30.074082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:50769)
0808 18:36:30.074351 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:30.074409 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260808 18:36:30.258530 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:30.258056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:55831)
0808 18:36:30.258373 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:30.258435 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260808 18:36:30.462589 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:30.461945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:33033)
0808 18:36:30.462412 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:30.462483 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
I20260808 18:36:31.075788 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:31.075307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:47983)
0808 18:36:31.075619 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:31.075680 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260808 18:36:31.259739 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:31.259292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:40217)
0808 18:36:31.259574 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:31.259635 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260808 18:36:31.463953 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:31.463437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:45889)
0808 18:36:31.463747 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:31.463827 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
I20260808 18:36:32.019076 12720 raft_consensus.cc:482] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:32.019233 12720 raft_consensus.cc:504] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e60d8245560480c8c80044501e178fc" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46373 } } peers { permanent_uuid: "e56f0e39bb224f5aaf0380719e86f4ff" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 46513 } } peers { permanent_uuid: "c314e40244c647968079c60718c0c802" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 41825 } }
I20260808 18:36:32.019666 12720 leader_election.cc:288] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373), e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513)
W20260808 18:36:32.020578 12285 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e56f0e39bb224f5aaf0380719e86f4ff (127.27.251.194:46513): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
W20260808 18:36:32.020711 12286 leader_election.cc:341] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e60d8245560480c8c80044501e178fc (127.27.251.195:46373): Not found: Tablet not found: 1865a0ffe9a2483aaca90026ecd2004b
I20260808 18:36:32.020788 12286 leader_election.cc:302] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c314e40244c647968079c60718c0c802; no voters: 5e60d8245560480c8c80044501e178fc, e56f0e39bb224f5aaf0380719e86f4ff
I20260808 18:36:32.020958 12720 raft_consensus.cc:2726] T 1865a0ffe9a2483aaca90026ecd2004b P c314e40244c647968079c60718c0c802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:36:32.077242 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:32.076776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:60423)
0808 18:36:32.077081 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:32.077143 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
I20260808 18:36:32.261024 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:32.260573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:53169)
0808 18:36:32.260866 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:32.260927 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260808 18:36:32.465410 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:32.464943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:40873)
0808 18:36:32.465240 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:32.465300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260808 18:36:33.078526 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:33.078073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:49249)
0808 18:36:33.078361 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:33.078418 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260808 18:36:33.262480 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:33.262079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:40571)
0808 18:36:33.262336 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:33.262388 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260808 18:36:33.466966 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:33.466357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:38635)
0808 18:36:33.466728 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:33.466824 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":83,"threads_started":1}
I20260808 18:36:34.079751 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:34.079293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:57011)
0808 18:36:34.079590 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:34.079649 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260808 18:36:34.263765 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:34.263252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:56111)
0808 18:36:34.263533 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:34.263594 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260808 18:36:34.468169 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:34.467745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:51167)
0808 18:36:34.468000 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:34.468065 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260808 18:36:35.081092 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:35.080632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:33007)
0808 18:36:35.080914 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:35.080987 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260808 18:36:35.265154 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:35.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:60779)
0808 18:36:35.264970 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:35.265038 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
I20260808 18:36:35.469648 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:35.469066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:48897)
0808 18:36:35.469405 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:35.469504 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260808 18:36:36.082325 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:36.081873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:48099)
0808 18:36:36.082160 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:36.082221 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260808 18:36:36.266486 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:36.266054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:37511)
0808 18:36:36.266323 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:36.266384 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260808 18:36:36.471084 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:36.470619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:41401)
0808 18:36:36.470923 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:36.470982 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260808 18:36:37.083601 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:37.083168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:56743)
0808 18:36:37.083434 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:37.083496 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260808 18:36:37.268031 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:37.267560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:53841)
0808 18:36:37.267861 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:37.267932 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260808 18:36:37.472584 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:37.472008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:40897)
0808 18:36:37.472356 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:37.472431 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260808 18:36:38.085376 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:38.084834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.193:56351)
0808 18:36:38.085177 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:38.085254 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":84,"threads_started":1}
I20260808 18:36:38.269616 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:38.269180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.194:36623)
0808 18:36:38.269451 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:38.269518 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260808 18:36:38.468700 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12231
I20260808 18:36:38.474157 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:38.473705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:41795 (local address 127.27.251.195:50845)
0808 18:36:38.473984 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:38.474047 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:41795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260808 18:36:38.477892 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12405
I20260808 18:36:38.485174 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12551
2026-08-08T18:36:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20420 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:36:39 dist-test-slave-fd7l krb5kdc[12745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:36:39 dist-test-slave-fd7l krb5kdc[12745](info): set up 2 sockets
Aug 08 18:36:39 dist-test-slave-fd7l krb5kdc[12745](info): commencing operation
krb5kdc: starting...
W20260808 18:36:40.338353 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:36:40 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:36:40.469054 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:40.579869 12759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:40.580176 12759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:40.580260 12759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:40.584530 12759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:36:40.584618 12759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:36:40.584645 12759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:36:40.584668 12759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:36:40.584707 12759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:40.584744 12759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:40.584800 12759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:36:40.584836 12759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:36:40.590130 12759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:40.591226 12759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:40.592286 12759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:40.600858 12764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:40.600909 12765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:40.600908 12768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:40.601051 12759 server_base.cc:1034] running on GCE node
I20260808 18:36:40.601689 12759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:40.602818 12759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:40.604007 12759 hybrid_clock.cc:648] HybridClock initialized: now 1786214200603979 us; error 38 us; skew 500 ppm
Aug 08 18:36:40 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214200, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:40.606745 12759 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:36:40.607671 12759 webserver.cc:460] Webserver started at http://127.27.251.254:38321/ using document root <none> and password file <none>
I20260808 18:36:40.608232 12759 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:40.608321 12759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:40.608614 12759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:40.610599 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "244b5b3c12f644708ad11aa3bd466313"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "dbe847a6270fa1ccb59b791c5f0f7981"
server_key_iv: "63911ebc4393ddf44f48c111c22007ad"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.611173 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "244b5b3c12f644708ad11aa3bd466313"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "dbe847a6270fa1ccb59b791c5f0f7981"
server_key_iv: "63911ebc4393ddf44f48c111c22007ad"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.614822 12759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260808 18:36:40.617225 12776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:40.618454 12759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:40.618594 12759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "244b5b3c12f644708ad11aa3bd466313"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "dbe847a6270fa1ccb59b791c5f0f7981"
server_key_iv: "63911ebc4393ddf44f48c111c22007ad"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.618716 12759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:40.658304 12759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:40.661665 12759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:40.661883 12759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:40.669425 12759 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:36251
I20260808 18:36:40.669780 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:36251 every 8 connection(s)
I20260808 18:36:40.670463 12759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:36:40.673645 12830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:40.676502 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12759
I20260808 18:36:40.676666 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:36:40.677032 28655 external_mini_cluster.cc:1428] Setting key f1c26d8c0d258be69fb15336752553ab
I20260808 18:36:40.680445 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap starting.
I20260808 18:36:40.683007 12830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:40.684008 12830 log.cc:824] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:40.686338 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: No bootstrap required, opened a new log
Aug 08 18:36:40 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:40.691675 12830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:36:40.691964 12830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:40.692036 12830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Initialized, Role: FOLLOWER
I20260808 18:36:40.692615 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:40.678628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.27.251.254:36251)
0808 18:36:40.679086 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:40.679109 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:40.679144 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:36:40.679775 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:40.679791 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:40.680126 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:40.680489 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:40.681368 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.682357 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.683245 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.684038 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.684227 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:40.686987 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:40.686993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:40.687009 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:40.687048 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:40.689330 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:40.689780 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:40.689787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:40.689793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:40.689952 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:40.690220 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:40.690224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:40.690226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:40.690775 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:40.690999 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:40.691209 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 18:36:40.691527 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260808 18:36:40.692574 12830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:36:40.692741 12830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:36:40.692811 12830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:36:40.692930 12830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:40.694126 12830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:36:40.694494 12830 leader_election.cc:302] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244b5b3c12f644708ad11aa3bd466313; no voters:
I20260808 18:36:40.694873 12830 leader_election.cc:288] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:36:40.694973 12835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:40.695245 12835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 LEADER]: Becoming Leader. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Running, Role: LEADER
I20260808 18:36:40.695628 12835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:36:40.696163 12830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:36:40.697351 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244b5b3c12f644708ad11aa3bd466313. Latest consensus state: current_term: 1 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:36:40.697489 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:40.697800 12836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:36:40.697932 12836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:36:40.698028 12845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:36:40.701093 12845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:36:40.706857 12845 catalog_manager.cc:1347] Generated new cluster ID: bb66fe4f43bc4d6185a7ece591a83f7a
I20260808 18:36:40.706957 12845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:36:40.720724 12845 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:36:40.721971 12845 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:36:40.729043 12845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Generated new TSK 0
I20260808 18:36:40.729758 12845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:36:40.761399 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:40.875857 12857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:40.876128 12857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:40.876171 12857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:40.880221 12857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:40.880276 12857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:40.880332 12857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:36:40.885558 12857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:40.886559 12857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:40.887562 12857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:40.897353 12863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:40.897337 12862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:40.897308 12865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:40.897601 12857 server_base.cc:1034] running on GCE node
I20260808 18:36:40.898005 12857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:40.898763 12857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:40.899983 12857 hybrid_clock.cc:648] HybridClock initialized: now 1786214200899964 us; error 26 us; skew 500 ppm
Aug 08 18:36:40 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214200, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:40.903110 12857 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:36:40.903945 12857 webserver.cc:460] Webserver started at http://127.27.251.193:41249/ using document root <none> and password file <none>
I20260808 18:36:40.904479 12857 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:40.904597 12857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:40.904779 12857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:40.906800 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "86b00ddf0f3a0be120184d0459223d45"
server_key_iv: "46e0e3bba105699a8e1354810fb6b5d5"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.907325 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "86b00ddf0f3a0be120184d0459223d45"
server_key_iv: "46e0e3bba105699a8e1354810fb6b5d5"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.911120 12857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 18:36:40.913738 12872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:40.915076 12857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:40.915212 12857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "86b00ddf0f3a0be120184d0459223d45"
server_key_iv: "46e0e3bba105699a8e1354810fb6b5d5"
server_key_version: "encryptionkey@0"
I20260808 18:36:40.915326 12857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:40.941345 12857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:40.944204 12857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:40.944409 12857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:40.945036 12857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:40.945869 12857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:40.945915 12857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:40.945986 12857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:40.946018 12857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:40.954823 12857 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37121
I20260808 18:36:40.954845 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37121 every 8 connection(s)
I20260808 18:36:40.955740 12857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:36:40.958964 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12857
I20260808 18:36:40.959105 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:36:40.959408 28655 external_mini_cluster.cc:1428] Setting key ac9a27f5251021cb0a32672e7308176f
Aug 08 18:36:40 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214200, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:36:40.971663 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:40.957601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:46575 (local address 127.27.251.254:36251)
0808 18:36:40.957773 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:40.957778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:40.958871 (+ 1093us) server_negotiation.cc:407] Connection header received
0808 18:36:40.959998 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:40.960001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:40.960056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:40.960152 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:40.961824 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.962692 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.964574 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.964823 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.964916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:40.967873 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:40.967877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:40.967879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:40.967903 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:40.969441 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:40.970035 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:40.970038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:40.970040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:40.970124 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:40.970496 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:40.970499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:40.970501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:40.970691 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:40.970800 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:40.971321 (+ 521us) server_negotiation.cc:299] Negotiation successful
0808 18:36:40.971481 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:36:40.972468 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:40.958044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:46575)
0808 18:36:40.958721 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:40.958762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:40.959721 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:40.960807 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:40.960821 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:40.961186 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:40.961670 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:40.961683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.962831 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.962833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:40.964324 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:40.964338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:40.964990 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:40.964997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:40.965256 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:40.965911 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:40.965940 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:40.967694 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:40.969569 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:40.969575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:40.969590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:40.969917 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:40.970216 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:40.970219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:40.970221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:40.970388 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:40.970800 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:40.970808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:40.971073 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 18:36:40.971277 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 18:36:40.971622 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:36:40.973855 12993 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:36:40.974112 12993 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:40.974622 12993 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:36:40.976243 12794 ts_manager.cc:194] Registered new tserver with Master: dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121)
I20260808 18:36:40.977555 12794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:46575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:36:40.988850 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:41.102734 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:41.102988 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:41.103030 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:41.107101 12998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:41.107157 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:41.107213 12998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:36:41.112228 12998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:41.113288 12998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:41.114259 12998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:41.124229 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:41.124411 12998 server_base.cc:1034] running on GCE node
W20260808 18:36:41.124243 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:41.124225 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:41.125028 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:41.125685 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:41.126883 12998 hybrid_clock.cc:648] HybridClock initialized: now 1786214201126867 us; error 29 us; skew 500 ppm
Aug 08 18:36:41 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214201, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:41.129767 12998 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:36:41.130599 12998 webserver.cc:460] Webserver started at http://127.27.251.194:43417/ using document root <none> and password file <none>
I20260808 18:36:41.131176 12998 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:41.131242 12998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:41.131417 12998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:41.133478 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "10f418cd91824eb1a0b8321ad7a5a317"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "1977edc1be4abc8840a569785d3fc4a1"
server_key_iv: "fec9d58e9f45a6639da3cbcb4152a894"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.134003 12998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "10f418cd91824eb1a0b8321ad7a5a317"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "1977edc1be4abc8840a569785d3fc4a1"
server_key_iv: "fec9d58e9f45a6639da3cbcb4152a894"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.137775 12998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:36:41.140085 13013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.141383 12998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:36:41.141479 12998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "10f418cd91824eb1a0b8321ad7a5a317"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "1977edc1be4abc8840a569785d3fc4a1"
server_key_iv: "fec9d58e9f45a6639da3cbcb4152a894"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.141564 12998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:41.154680 12998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:41.157788 12998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:41.157955 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:41.158504 12998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:41.159286 12998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:41.159332 12998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.159370 12998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:41.159385 12998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.169251 12998 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:45615
I20260808 18:36:41.169287 13134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:45615 every 8 connection(s)
I20260808 18:36:41.170151 12998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:36:41.175318 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 12998
I20260808 18:36:41.175464 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:36:41.175853 28655 external_mini_cluster.cc:1428] Setting key 335dc7eb946096a26a8f43527715ee8b
Aug 08 18:36:41 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214201, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:41.185674 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.171992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:59509 (local address 127.27.251.254:36251)
0808 18:36:41.172144 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.172148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.173235 (+ 1087us) server_negotiation.cc:407] Connection header received
0808 18:36:41.174072 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.174074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.174121 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.174196 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:41.175407 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.176206 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.177684 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.178087 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.178222 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.181297 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:41.181303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:41.181309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:41.181344 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:41.183146 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.183778 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.183785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.183789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.183884 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.184260 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.184265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.184268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.184565 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:41.184683 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.185343 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.185481 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:36:41.186532 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.172376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:59509)
0808 18:36:41.173078 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.173124 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.173874 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.174365 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.174372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.174739 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:41.175248 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.175261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.176358 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.176364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.177544 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.177554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.178292 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.178295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.178444 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.179146 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:41.179164 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:41.181149 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:41.183323 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:41.183329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:41.183341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:41.183649 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:41.183986 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:41.183988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:41.183990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:41.184146 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:41.184698 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:41.184704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:41.185023 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.185234 (+ 211us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.185520 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:36:41.187916 13136 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:36:41.188160 13136 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:41.188686 13136 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:36:41.189795 12794 ts_manager.cc:194] Registered new tserver with Master: 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:36:41.190532 12794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:59509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:36:41.207762 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:36:41.321596 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:36:41.321868 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:36:41.321946 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:36:41.326200 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:36:41.326290 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:36:41.326383 13141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:36:41.331655 13141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:36:41.332777 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:36:41.333841 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:36:41.343745 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:41.343986 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:36:41.343988 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:36:41.344180 13141 server_base.cc:1034] running on GCE node
I20260808 18:36:41.344776 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:36:41.345513 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:36:41.346715 13141 hybrid_clock.cc:648] HybridClock initialized: now 1786214201346696 us; error 29 us; skew 500 ppm
Aug 08 18:36:41 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214201, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:36:41.349680 13141 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:36:41.350574 13141 webserver.cc:460] Webserver started at http://127.27.251.195:45087/ using document root <none> and password file <none>
I20260808 18:36:41.351189 13141 fs_manager.cc:359] Metadata directory not provided
I20260808 18:36:41.351289 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:36:41.351542 13141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:36:41.353806 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "bbbdfb52d582428a8c3e44886167ed1f"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "8e76066c7d0395eab892d02d014b8b28"
server_key_iv: "02b0da26b3f5513fb13ef963d5df2525"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.354511 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "bbbdfb52d582428a8c3e44886167ed1f"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "8e76066c7d0395eab892d02d014b8b28"
server_key_iv: "02b0da26b3f5513fb13ef963d5df2525"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.358705 13141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:36:41.361608 13156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.362941 13141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:36:41.363056 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "bbbdfb52d582428a8c3e44886167ed1f"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "8e76066c7d0395eab892d02d014b8b28"
server_key_iv: "02b0da26b3f5513fb13ef963d5df2525"
server_key_version: "encryptionkey@0"
I20260808 18:36:41.363204 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:36:41.377281 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:36:41.380214 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:36:41.380420 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:36:41.381038 13141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:36:41.381898 13141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:36:41.381944 13141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.382016 13141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:36:41.382047 13141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:36:41.390990 13141 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:36557
I20260808 18:36:41.391011 13274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:36557 every 8 connection(s)
I20260808 18:36:41.391870 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:36:41.394903 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13141
I20260808 18:36:41.395035 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:36:41.395356 28655 external_mini_cluster.cc:1428] Setting key a45c2c465729bfc092b8fa072b61a102
Aug 08 18:36:41 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214201, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:36:41.405941 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.393653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:34573 (local address 127.27.251.254:36251)
0808 18:36:41.393805 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.393809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.394868 (+ 1059us) server_negotiation.cc:407] Connection header received
0808 18:36:41.396078 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.396081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.396135 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.396223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:41.397775 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.398321 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.399678 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.399885 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.399959 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.402514 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:41.402517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:41.402519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:41.402544 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:41.403845 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.404302 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.404305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.404306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.404379 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.404834 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.404837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.404839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.405035 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:41.405172 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.405654 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.405775 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:36:41.406762 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.393989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:34573)
0808 18:36:41.394686 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.394746 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.395835 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.396432 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.396441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.397001 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:36:41.397589 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.397612 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.398449 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.398452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.399553 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.399574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.400059 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.400063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.400232 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.400796 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:36:41.400814 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:36:41.402306 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:36:41.403951 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:41.403956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:41.403966 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:41.404216 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:41.404546 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:36:41.404552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:36:41.404554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:36:41.404716 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:36:41.405171 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:36:41.405178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:36:41.405415 (+ 237us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.405594 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.405916 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":113,"threads_started":1}
I20260808 18:36:41.408218 13275 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:36:41.408595 13275 heartbeater.cc:464] Registering TS with master...
I20260808 18:36:41.409245 13275 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:36:41.410367 12794 ts_manager.cc:194] Registered new tserver with Master: bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557)
I20260808 18:36:41.410923 12794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:34573
I20260808 18:36:41.421188 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:36:41.430605 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.423374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.27.251.254:36251)
0808 18:36:41.423490 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.423493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.423739 (+ 246us) server_negotiation.cc:407] Connection header received
0808 18:36:41.423834 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.423837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.423873 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.423952 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:36:41.424717 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.425214 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.426019 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.426231 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.426303 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.427619 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:36:41.427621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:36:41.427624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:36:41.427645 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:36:41.428905 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.429332 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.429334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.429335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.429405 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:36:41.429730 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:36:41.429734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:36:41.429735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:36:41.429979 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:36:41.430134 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.430370 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.430476 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:36:41.433531 12794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:36:41.436475 12794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:36:41.451435 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.445173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:45615 (local address 127.0.0.1:37804)
0808 18:36:41.445606 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.445663 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.445888 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.446701 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.446707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.446732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:41.446869 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.446876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.448318 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.448322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.449925 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.449936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.451028 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.451031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.451113 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.451144 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.451240 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.451324 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":41,"thread_start_us":88,"threads_started":1}
I20260808 18:36:41.451586 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.445303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.27.251.194:45615)
0808 18:36:41.445734 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.445748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.445841 (+ 93us) server_negotiation.cc:407] Connection header received
0808 18:36:41.446098 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.446105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.446321 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.446493 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:41.447011 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.448173 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.450065 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.450842 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.451022 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.451163 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.451292 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.451396 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":190,"threads_started":1}
I20260808 18:36:41.452231 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.446329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:37121 (local address 127.0.0.1:33820)
0808 18:36:41.447177 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.447189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.447282 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.447718 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.447723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.447742 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:41.447834 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.447840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.449107 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.449110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.450309 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.450321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.451776 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.451785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.451906 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.451934 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.452030 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.452104 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":33,"thread_start_us":100,"threads_started":1}
I20260808 18:36:41.452332 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.446655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.27.251.193:37121)
0808 18:36:41.447104 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.447118 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.447287 (+ 169us) server_negotiation.cc:407] Connection header received
0808 18:36:41.447391 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.447396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.447561 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.447697 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:41.447971 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.448969 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.450455 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.451406 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.451551 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.451709 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.452068 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.452164 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":131,"threads_started":1}
I20260808 18:36:41.453514 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.445976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:36557 (local address 127.0.0.1:37920)
0808 18:36:41.446817 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.446831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.446964 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.447453 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.447456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.447475 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:41.447571 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.447579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.448937 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.448940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.450309 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.450321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.452860 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.452864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.453217 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.453248 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.453305 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.453384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":54,"thread_start_us":97,"threads_started":1}
I20260808 18:36:41.454103 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.446167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.27.251.195:36557)
0808 18:36:41.446653 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.446663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.446938 (+ 275us) server_negotiation.cc:407] Connection header received
0808 18:36:41.447088 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.447095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.447302 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.447468 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:41.447676 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.448789 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.451781 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.452705 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.453756 (+ 1051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.453868 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.453913 (+ 45us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.453985 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":98,"threads_started":1}
I20260808 18:36:41.454877 13063 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e68de7754438497b01a1359b3844d (DEFAULT_TABLE table=test-table [id=77cf16338434415fb00c7b3bcb90a136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:41.455240 12923 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e68de7754438497b01a1359b3844d (DEFAULT_TABLE table=test-table [id=77cf16338434415fb00c7b3bcb90a136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:41.456055 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e68de7754438497b01a1359b3844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:41.456202 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e68de7754438497b01a1359b3844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:41.457855 13205 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e68de7754438497b01a1359b3844d (DEFAULT_TABLE table=test-table [id=77cf16338434415fb00c7b3bcb90a136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:36:41.458657 13205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e68de7754438497b01a1359b3844d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:36:41.464546 13292 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Bootstrap starting.
I20260808 18:36:41.465826 13293 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap starting.
I20260808 18:36:41.467585 13295 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Bootstrap starting.
I20260808 18:36:41.467689 13292 tablet_bootstrap.cc:654] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:41.468683 13292 log.cc:824] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:41.469811 13295 tablet_bootstrap.cc:654] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:41.470091 13293 tablet_bootstrap.cc:654] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Neither blocks nor log segments found. Creating new log.
I20260808 18:36:41.470949 13295 log.cc:824] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:41.471145 13293 log.cc:824] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Log is configured to *not* fsync() on all Append() calls
I20260808 18:36:41.472771 13292 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: No bootstrap required, opened a new log
I20260808 18:36:41.473044 13292 ts_tablet_manager.cc:1397] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260808 18:36:41.473169 13293 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: No bootstrap required, opened a new log
I20260808 18:36:41.473439 13293 ts_tablet_manager.cc:1397] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 18:36:41.473803 13295 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: No bootstrap required, opened a new log
I20260808 18:36:41.473984 13295 ts_tablet_manager.cc:1397] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:36:41.476209 13292 raft_consensus.cc:348] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.476480 13292 raft_consensus.cc:374] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:41.476589 13292 raft_consensus.cc:729] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f418cd91824eb1a0b8321ad7a5a317, State: Initialized, Role: FOLLOWER
I20260808 18:36:41.477087 13292 consensus_queue.cc:260] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.477356 13293 raft_consensus.cc:348] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.477685 13293 raft_consensus.cc:374] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:41.477783 13293 raft_consensus.cc:729] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbf0b5f2c104c17ae7b08f768e674cf, State: Initialized, Role: FOLLOWER
I20260808 18:36:41.477795 13136 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
I20260808 18:36:41.477770 13295 raft_consensus.cc:348] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.478072 13295 raft_consensus.cc:374] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:36:41.478163 13295 raft_consensus.cc:729] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbdfb52d582428a8c3e44886167ed1f, State: Initialized, Role: FOLLOWER
I20260808 18:36:41.478225 13292 ts_tablet_manager.cc:1428] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260808 18:36:41.478416 13293 consensus_queue.cc:260] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.478612 13295 consensus_queue.cc:260] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.479107 12993 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
I20260808 18:36:41.479398 13293 ts_tablet_manager.cc:1428] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:36:41.479501 13295 ts_tablet_manager.cc:1428] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260808 18:36:41.479866 13275 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
I20260808 18:36:41.507616 13299 raft_consensus.cc:482] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:36:41.507784 13299 raft_consensus.cc:504] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.508960 13299 leader_election.cc:288] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121)
I20260808 18:36:41.512619 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.509298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36909 (local address 127.27.251.195:36557)
0808 18:36:41.509425 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.509446 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.509467 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:36:41.509529 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.509532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.509579 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.509675 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:41.509975 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.510519 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.511547 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.511956 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.512083 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.512137 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.512382 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.512470 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:36:41.512617 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.509191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:36557 (local address 127.27.251.194:36909)
0808 18:36:41.509327 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.509343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.509436 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.509707 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.509710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.509731 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:41.509849 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.509854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.510635 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.510639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.511413 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.511422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.512083 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.512087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.512224 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.512243 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.512347 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.512450 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260808 18:36:41.513273 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.509648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:49641 (local address 127.27.251.193:37121)
0808 18:36:41.509773 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.509778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.509959 (+ 181us) server_negotiation.cc:407] Connection header received
0808 18:36:41.510090 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.510093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.510133 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.510217 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:36:41.510494 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.510977 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.511992 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.512587 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.512710 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.512769 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.513090 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.513161 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:36:41.513276 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.509495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:37121 (local address 127.27.251.194:49641)
0808 18:36:41.509867 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.509878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:36:41.509997 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:36:41.510226 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:36:41.510229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:36:41.510244 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:36:41.510375 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.510380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.511086 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.511091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.511865 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:36:41.511874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.512856 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.512861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:36:41.512969 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.512991 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 18:36:41.513065 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 18:36:41.513135 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":108,"threads_started":1}
I20260808 18:36:41.513270 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e68de7754438497b01a1359b3844d" candidate_uuid: "10f418cd91824eb1a0b8321ad7a5a317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbdfb52d582428a8c3e44886167ed1f" is_pre_election: true
I20260808 18:36:41.513610 13227 raft_consensus.cc:2454] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f418cd91824eb1a0b8321ad7a5a317 in term 0.
I20260808 18:36:41.513931 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e68de7754438497b01a1359b3844d" candidate_uuid: "10f418cd91824eb1a0b8321ad7a5a317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" is_pre_election: true
I20260808 18:36:41.514129 13018 leader_election.cc:302] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f; no voters:
I20260808 18:36:41.514247 12946 raft_consensus.cc:2454] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f418cd91824eb1a0b8321ad7a5a317 in term 0.
I20260808 18:36:41.514428 13299 raft_consensus.cc:2781] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:36:41.514545 13299 raft_consensus.cc:482] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:36:41.514590 13299 raft_consensus.cc:3037] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:41.515611 13299 raft_consensus.cc:504] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.515926 13299 leader_election.cc:288] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [CANDIDATE]: Term 1 election: Requested vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121)
I20260808 18:36:41.516240 13227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e68de7754438497b01a1359b3844d" candidate_uuid: "10f418cd91824eb1a0b8321ad7a5a317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbdfb52d582428a8c3e44886167ed1f"
I20260808 18:36:41.516338 12946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e68de7754438497b01a1359b3844d" candidate_uuid: "10f418cd91824eb1a0b8321ad7a5a317" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
I20260808 18:36:41.516453 12946 raft_consensus.cc:3037] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:41.516371 13227 raft_consensus.cc:3037] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:36:41.517791 13227 raft_consensus.cc:2454] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f418cd91824eb1a0b8321ad7a5a317 in term 1.
I20260808 18:36:41.517791 12946 raft_consensus.cc:2454] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f418cd91824eb1a0b8321ad7a5a317 in term 1.
I20260808 18:36:41.518096 13018 leader_election.cc:302] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f418cd91824eb1a0b8321ad7a5a317, dbbf0b5f2c104c17ae7b08f768e674cf; no voters:
I20260808 18:36:41.518260 13299 raft_consensus.cc:2781] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:36:41.518515 13299 raft_consensus.cc:686] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 LEADER]: Becoming Leader. State: Replica: 10f418cd91824eb1a0b8321ad7a5a317, State: Running, Role: LEADER
I20260808 18:36:41.518918 13299 consensus_queue.cc:237] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:36:41.522410 12794 catalog_manager.cc:5495] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "10f418cd91824eb1a0b8321ad7a5a317" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } health_report { overall_health: UNKNOWN } } }
I20260808 18:36:41.536213 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.532545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.27.251.194:45615)
0808 18:36:41.532659 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:36:41.532664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:36:41.532757 (+ 93us) server_negotiation.cc:407] Connection header received
0808 18:36:41.532945 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:36:41.532947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:36:41.533022 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:36:41.533134 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:36:41.533391 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.533895 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.534793 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:36:41.535014 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:36:41.535118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:36:41.535440 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:36:41.535838 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:36:41.535981 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 18:36:41.536024 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 18:36:41.536078 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 18:36:41.542244 13227 raft_consensus.cc:1261] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Refusing update from remote peer 10f418cd91824eb1a0b8321ad7a5a317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:41.542272 12946 raft_consensus.cc:1261] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Refusing update from remote peer 10f418cd91824eb1a0b8321ad7a5a317: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:36:41.543007 13303 consensus_queue.cc:1035] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:41.543192 13299 consensus_queue.cc:1035] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:36:41.551215 13309 mvcc.cc:205] Tried to move back new op lower bound from 7316333369508319232 to 7316333369424314368. Current Snapshot: MvccSnapshot[applied={T|T < 7316333369508319232}]
I20260808 18:36:41.552173 13308 mvcc.cc:205] Tried to move back new op lower bound from 7316333369508319232 to 7316333369424314368. Current Snapshot: MvccSnapshot[applied={T|T < 7316333369508319232}]
I20260808 18:36:41.560118 12794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:41.560261 12794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:36:41.562718 12794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Sending DeleteTablet for 3 replicas of tablet ec6e68de7754438497b01a1359b3844d
I20260808 18:36:41.563475 13063 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6e68de7754438497b01a1359b3844d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37804
I20260808 18:36:41.563617 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12857
I20260808 18:36:41.563666 13205 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6e68de7754438497b01a1359b3844d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37920
I20260808 18:36:41.565886 13317 tablet_replica.cc:330] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: stopping tablet replica
I20260808 18:36:41.566285 13317 raft_consensus.cc:2229] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:36:41.566581 13319 tablet_replica.cc:330] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: stopping tablet replica
I20260808 18:36:41.566663 13317 raft_consensus.cc:2258] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:41.566910 13319 raft_consensus.cc:2229] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:36:41.567363 13319 raft_consensus.cc:2258] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:36:41.568961 13317 ts_tablet_manager.cc:1905] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:41.569381 13319 ts_tablet_manager.cc:1905] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:36:41.571858 13317 ts_tablet_manager.cc:1918] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:41.572042 13317 log.cc:1196] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/ec6e68de7754438497b01a1359b3844d
I20260808 18:36:41.572480 13317 ts_tablet_manager.cc:1939] T ec6e68de7754438497b01a1359b3844d P bbbdfb52d582428a8c3e44886167ed1f: Deleting consensus metadata
I20260808 18:36:41.572947 13319 ts_tablet_manager.cc:1918] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:36:41.573132 13319 log.cc:1196] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/ec6e68de7754438497b01a1359b3844d
I20260808 18:36:41.573560 13319 ts_tablet_manager.cc:1939] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Deleting consensus metadata
I20260808 18:36:41.573845 12780 catalog_manager.cc:4841] TS bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): tablet ec6e68de7754438497b01a1359b3844d (table test-table [id=77cf16338434415fb00c7b3bcb90a136]) successfully deleted
W20260808 18:36:41.574088 12780 connection.cc:535] client connection to 127.27.251.193:37121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 18:36:41.574225 12780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:36:41.574368 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12998
I20260808 18:36:41.578251 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.577884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:37121 (local address 127.0.0.1:33824)
0808 18:36:41.578080 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.578150 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260808 18:36:41.578437 12780 catalog_manager.cc:4584] TS dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec6e68de7754438497b01a1359b3844d: Network error: Client connection negotiation failed: client connection to 127.27.251.193:37121: connect: Connection refused (error 111)
I20260808 18:36:41.584431 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13141
I20260808 18:36:41.586630 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:36:41.584759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:45615 (local address 127.0.0.1:37822)
0808 18:36:41.586453 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0808 18:36:41.586524 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.194:45615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1584}
I20260808 18:36:41.591496 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 12759
I20260808 18:37:01.600589 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--webserver_interface=127.27.251.254
--webserver_port=38321
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:01.714637 13321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:01.714882 13321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:01.714922 13321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:01.719002 13321 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:37:01.719045 13321 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:37:01.719061 13321 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:37:01.719075 13321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:01.719130 13321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:01.719164 13321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:01.719208 13321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:01.719255 13321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:01.724677 13321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:01.725682 13321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:01.726661 13321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:01.741385 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:01.741364 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:01.741372 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:01.741457 13321 server_base.cc:1034] running on GCE node
I20260808 18:37:01.742260 13321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:01.743410 13321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:01.744606 13321 hybrid_clock.cc:648] HybridClock initialized: now 1786214221744584 us; error 30 us; skew 500 ppm
Aug 08 18:37:01 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214221, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:01.753378 13321 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:37:01.754364 13321 webserver.cc:460] Webserver started at http://127.27.251.254:38321/ using document root <none> and password file <none>
I20260808 18:37:01.754998 13321 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:01.755115 13321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:01.759143 13321 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260808 18:37:01.761372 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:01.762735 13321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:37:01.762889 13321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "244b5b3c12f644708ad11aa3bd466313"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "dbe847a6270fa1ccb59b791c5f0f7981"
server_key_iv: "63911ebc4393ddf44f48c111c22007ad"
server_key_version: "encryptionkey@0"
I20260808 18:37:01.763500 13321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:01.779830 13321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:01.784273 13321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:01.784555 13321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:01.794597 13321 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:36251
I20260808 18:37:01.794904 13392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:36251 every 8 connection(s)
I20260808 18:37:01.796188 13321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:01.798578 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13321
I20260808 18:37:01.799091 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:37121
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=41249
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:01.801827 13393 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:37:01.803671 13393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap starting.
I20260808 18:37:01.827612 13393 log.cc:824] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:01.838738 13393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:37:01.839146 13393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap complete.
I20260808 18:37:01.843465 13393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:01.843854 13393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Initialized, Role: FOLLOWER
I20260808 18:37:01.844336 13393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:01.844466 13393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:01.844594 13393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:01.844710 13393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 1 FOLLOWER]: Advancing to term 2
I20260808 18:37:01.846621 13393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:01.846963 13393 leader_election.cc:302] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244b5b3c12f644708ad11aa3bd466313; no voters:
I20260808 18:37:01.847307 13393 leader_election.cc:288] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 18:37:01.847391 13397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 18:37:01.847597 13397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 LEADER]: Becoming Leader. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Running, Role: LEADER
I20260808 18:37:01.847960 13397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:01.848598 13393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:01.848945 13397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244b5b3c12f644708ad11aa3bd466313. Latest consensus state: current_term: 2 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:37:01.849159 13397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:01.849354 13398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:37:01.849476 13398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:01.849870 13402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:01.854867 13402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:01.855877 13402 catalog_manager.cc:1259] Loaded cluster ID: bb66fe4f43bc4d6185a7ece591a83f7a
I20260808 18:37:01.855940 13402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:01.858484 13402 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:01.859231 13402 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Loaded TSK: 0
I20260808 18:37:01.872809 13402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Generated new TSK 1
I20260808 18:37:01.873088 13402 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:37:01.926563 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:01.926863 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:01.926955 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:01.930994 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:01.931077 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:01.931169 13395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:01.936295 13395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=41249
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:01.937502 13395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:01.938495 13395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:01.949527 13424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:01.949532 13423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:01.949527 13426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:01.950115 13395 server_base.cc:1034] running on GCE node
I20260808 18:37:01.950523 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:01.951170 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:01.952368 13395 hybrid_clock.cc:648] HybridClock initialized: now 1786214221952341 us; error 43 us; skew 500 ppm
Aug 08 18:37:01 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214221, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:01.955423 13395 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:37:01.956358 13395 webserver.cc:460] Webserver started at http://127.27.251.193:41249/ using document root <none> and password file <none>
I20260808 18:37:01.956974 13395 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:01.957094 13395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:01.961170 13395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:01.963542 13434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:01.964911 13395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:01.965047 13395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "86b00ddf0f3a0be120184d0459223d45"
server_key_iv: "46e0e3bba105699a8e1354810fb6b5d5"
server_key_version: "encryptionkey@0"
I20260808 18:37:01.965627 13395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:01.976369 13395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:01.979604 13395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:01.979807 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:01.980401 13395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:01.981609 13441 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:37:01.985697 13395 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:37:01.985773 13395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260808 18:37:01.985843 13395 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:37:01.987488 13395 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:37:01.987545 13395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 18:37:01.987838 13441 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap starting.
I20260808 18:37:01.998783 13395 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37121
I20260808 18:37:01.998847 13555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37121 every 8 connection(s)
I20260808 18:37:02.000154 13395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:02.007584 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13395
I20260808 18:37:02.008492 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:45615
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=43417
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:02.010421 13441 log.cc:824] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214221, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:02.020460 13441 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:37:02.021102 13441 tablet_bootstrap.cc:492] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap complete.
I20260808 18:37:02.021680 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.002453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:47845)
0808 18:37:02.003206 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.003267 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.004199 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.005756 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.005764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.006188 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:02.006741 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.006758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.008833 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.008837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.010123 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.010135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.011155 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.011161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.011398 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.012206 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:02.012232 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:02.014432 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:02.017524 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.017531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.017547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.017939 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.018422 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.018427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.018431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.018629 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.019441 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:02.019451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:02.019797 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.020031 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.020426 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":160,"threads_started":1}
I20260808 18:37:02.021704 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.002433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:47845 (local address 127.27.251.254:36251)
0808 18:37:02.002858 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.002879 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.003391 (+ 512us) server_negotiation.cc:407] Connection header received
0808 18:37:02.004713 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.004737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.005269 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.005651 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.006948 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.008641 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.010293 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.010968 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.011198 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.014690 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.014700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.014718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.014757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.017354 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.018091 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.018098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.018104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.018284 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.018756 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.018760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.018762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.019278 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.019432 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.020199 (+ 767us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.020598 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":130,"threads_started":1}
I20260808 18:37:02.022830 13441 ts_tablet_manager.cc:1397] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.008s
I20260808 18:37:02.025112 13556 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:37:02.025350 13556 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:02.025887 13556 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:37:02.026046 13441 raft_consensus.cc:348] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.026541 13441 raft_consensus.cc:729] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbf0b5f2c104c17ae7b08f768e674cf, State: Initialized, Role: FOLLOWER
I20260808 18:37:02.027137 13441 consensus_queue.cc:260] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.028258 13441 ts_tablet_manager.cc:1428] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:02.030263 13356 ts_manager.cc:194] Registered new tserver with Master: dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121)
I20260808 18:37:02.034049 13356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:47845
I20260808 18:37:02.036798 13556 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214221, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.193@KRBTEST.COM
I20260808 18:37:02.045543 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.033510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.27.251.193:37121)
0808 18:37:02.034355 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.034370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.034397 (+ 27us) server_negotiation.cc:407] Connection header received
0808 18:37:02.034469 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.034474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.034670 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.034796 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.035639 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.037099 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.038314 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.038800 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.038893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.041602 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.041612 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.041622 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.041646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.043381 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.043951 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.043955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.043956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.044037 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.044459 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.044462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.044464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.044777 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.044883 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.045267 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.045409 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":104,"threads_started":1}
I20260808 18:37:02.045580 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.033398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:37121 (local address 127.0.0.1:57744)
0808 18:37:02.034024 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.034073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.034218 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.034962 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.034969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.035377 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:02.035500 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.035508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.037254 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.037258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.038182 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.038194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.038924 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.038928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.039041 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.039657 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:02.039673 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:02.041442 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:02.043519 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.043529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.043534 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.043833 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.044127 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.044130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.044132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.044231 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.044892 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:02.044898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:02.045030 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.045189 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.045393 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":109,"threads_started":1}
I20260808 18:37:02.046679 13486 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6e68de7754438497b01a1359b3844d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:36:41 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57744
I20260808 18:37:02.047600 13566 tablet_replica.cc:330] stopping tablet replica
I20260808 18:37:02.047972 13566 raft_consensus.cc:2229] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:02.048353 13566 raft_consensus.cc:2258] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:02.049682 13566 ts_tablet_manager.cc:1905] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:02.054873 13566 ts_tablet_manager.cc:1918] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:02.054987 13566 log.cc:1196] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal/wals/ec6e68de7754438497b01a1359b3844d
I20260808 18:37:02.055282 13566 ts_tablet_manager.cc:1939] T ec6e68de7754438497b01a1359b3844d P dbbf0b5f2c104c17ae7b08f768e674cf: Deleting consensus metadata
I20260808 18:37:02.057525 13342 catalog_manager.cc:4841] TS dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121): tablet ec6e68de7754438497b01a1359b3844d (table test-table [id=77cf16338434415fb00c7b3bcb90a136]) successfully deleted
W20260808 18:37:02.136013 13560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:02.136266 13560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:02.136308 13560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:02.140405 13560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:02.140461 13560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:02.140576 13560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:02.146083 13560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=43417
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:02.147035 13560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:02.147967 13560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:02.158084 13571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:02.158154 13560 server_base.cc:1034] running on GCE node
W20260808 18:37:02.158092 13574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:02.158087 13572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:02.158787 13560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:02.159524 13560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:02.160684 13560 hybrid_clock.cc:648] HybridClock initialized: now 1786214222160669 us; error 24 us; skew 500 ppm
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214222, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:02.163563 13560 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:37:02.164469 13560 webserver.cc:460] Webserver started at http://127.27.251.194:43417/ using document root <none> and password file <none>
I20260808 18:37:02.165055 13560 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:02.165179 13560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:02.169350 13560 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:37:02.171608 13581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:02.172951 13560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:37:02.173089 13560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "10f418cd91824eb1a0b8321ad7a5a317"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "1977edc1be4abc8840a569785d3fc4a1"
server_key_iv: "fec9d58e9f45a6639da3cbcb4152a894"
server_key_version: "encryptionkey@0"
I20260808 18:37:02.173717 13560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:02.197857 13560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:02.201462 13560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:02.201696 13560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:02.202400 13560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:02.203545 13588 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260808 18:37:02.207804 13588 ts_tablet_manager.cc:1852] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260808 18:37:02.208237 13588 ts_tablet_manager.cc:1905] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:02.212719 13588 ts_tablet_manager.cc:1918] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:02.212913 13588 ts_tablet_manager.cc:1939] T ec6e68de7754438497b01a1359b3844d P 10f418cd91824eb1a0b8321ad7a5a317: Deleting consensus metadata
I20260808 18:37:02.216022 13560 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260808 18:37:02.216138 13560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.000s
I20260808 18:37:02.216253 13560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:02.216315 13560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:02.226544 13560 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:45615
I20260808 18:37:02.226581 13701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:45615 every 8 connection(s)
I20260808 18:37:02.227486 13560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214222, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:02.237787 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13560
I20260808 18:37:02.238659 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:36557
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=45087
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:02.241946 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.229560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:43717 (local address 127.27.251.254:36251)
0808 18:37:02.229715 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.229719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.230562 (+ 843us) server_negotiation.cc:407] Connection header received
0808 18:37:02.231474 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.231477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.231525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.231607 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.232786 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.233393 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.234569 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.234826 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.234940 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.237641 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.237646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.237649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.237678 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.239476 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.239966 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.239973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.239977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.240078 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.240435 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.240441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.240446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.240742 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.240862 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.241588 (+ 726us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.241750 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:37:02.243079 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.229914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:43717)
0808 18:37:02.230385 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.230428 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.231261 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.231767 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.231774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.232136 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:02.232638 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.232652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.233566 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.233568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.234424 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.234434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.234979 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.234982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.235149 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.235715 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:02.235730 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:02.237393 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:02.239601 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.239605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.239616 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.239866 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.240182 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.240184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.240186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.240320 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.240895 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:02.240905 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:02.241284 (+ 379us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.241550 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.241989 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260808 18:37:02.244491 13702 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:37:02.244792 13702 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:02.245301 13702 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:37:02.246402 13356 ts_manager.cc:194] Registered new tserver with Master: 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
I20260808 18:37:02.246989 13356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:43717
W20260808 18:37:02.355901 13705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:02.356174 13705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:02.356256 13705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:02.360256 13705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:02.360343 13705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:02.360433 13705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:02.365628 13705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:36557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=45087
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:02.366701 13705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:02.367805 13705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:02.377501 13710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:02.377501 13713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:02.377816 13711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:02.377815 13705 server_base.cc:1034] running on GCE node
I20260808 18:37:02.378288 13705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:02.378906 13705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:02.380069 13705 hybrid_clock.cc:648] HybridClock initialized: now 1786214222380055 us; error 26 us; skew 500 ppm
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214222, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:02.382848 13705 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:37:02.383785 13705 webserver.cc:460] Webserver started at http://127.27.251.195:45087/ using document root <none> and password file <none>
I20260808 18:37:02.384671 13705 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:02.384796 13705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:02.388785 13705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:37:02.390946 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:02.392253 13705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:37:02.392429 13705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "bbbdfb52d582428a8c3e44886167ed1f"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "8e76066c7d0395eab892d02d014b8b28"
server_key_iv: "02b0da26b3f5513fb13ef963d5df2525"
server_key_version: "encryptionkey@0"
I20260808 18:37:02.393301 13705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:02.404314 13705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:02.407519 13705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:02.407775 13705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:02.408390 13705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:02.409294 13705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:02.409368 13705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:02.409436 13705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:02.409484 13705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:02.419188 13705 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:36557
I20260808 18:37:02.419212 13845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:36557 every 8 connection(s)
I20260808 18:37:02.420183 13705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:02.427270 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13705
Aug 08 18:37:02 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214222, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:02.434476 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.422113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:39229 (local address 127.27.251.254:36251)
0808 18:37:02.422286 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.422290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.423477 (+ 1187us) server_negotiation.cc:407] Connection header received
0808 18:37:02.424329 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.424333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.424381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.424462 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.425686 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.426228 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.427375 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.427699 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.427808 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.430540 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.430543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.430545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.430568 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.432290 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.432881 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.432885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.432887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.432980 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.433343 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.433346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.433348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.433544 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.433655 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.434164 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.434281 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"spinlock_wait_cycles":23296}
I20260808 18:37:02.435333 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.422576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:39229)
0808 18:37:02.423322 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.423374 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.424140 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.424697 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.424703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.425075 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:02.425553 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.425567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.426349 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.426352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.427243 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.427253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.427870 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.427876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.428086 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.428816 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:02.428839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:02.430368 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:02.432413 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.432419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.432434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.432771 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.433078 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:02.433082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:02.433083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:02.433238 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:02.433655 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:02.433662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:02.433906 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.434095 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.434416 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":112,"threads_started":1}
I20260808 18:37:02.436720 13846 heartbeater.cc:347] Connected to a master server at 127.27.251.254:36251
I20260808 18:37:02.436985 13846 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:02.437458 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.428038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.27.251.254:36251)
0808 18:37:02.428255 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.428260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.428495 (+ 235us) server_negotiation.cc:407] Connection header received
0808 18:37:02.428665 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.428667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.428735 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.428833 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.429671 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.430345 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.431626 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.432030 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.432157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.433743 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.433747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.433750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.433786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.435525 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.436023 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.436025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.436027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.436141 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.436446 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.436449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.436450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.436684 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.436901 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.437175 (+ 274us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.437295 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260808 18:37:02.437587 13846 heartbeater.cc:511] Master 127.27.251.254:36251 requested a full tablet report, sending...
I20260808 18:37:02.438799 13355 ts_manager.cc:194] Registered new tserver with Master: bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557)
I20260808 18:37:02.439533 13355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:39229
I20260808 18:37:02.451319 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:02.456009 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.452058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.27.251.254:36251)
0808 18:37:02.452263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.452268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.452437 (+ 169us) server_negotiation.cc:407] Connection header received
0808 18:37:02.452700 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.452702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.452808 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.452950 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:02.453219 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.453755 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.454688 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.454929 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.455046 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.455390 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:02.455552 (+ 162us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 18:37:02.455862 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58458: token expired
Metrics: {"server-negotiator.queue_time_us":107}
W20260808 18:37:02.456111 13849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58458: token expired
W20260808 18:37:02.456383 13281 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.251.254:36251)
I20260808 18:37:02.464718 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.456849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.27.251.254:36251)
0808 18:37:02.456995 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.457000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.457021 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:37:02.457110 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.457113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.457159 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.457241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:02.457916 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.458403 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.459211 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.459435 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.459548 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.461178 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:02.461181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:02.461184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:02.461211 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:02.462747 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.463252 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.463257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.463259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.463357 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:02.463731 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:02.463737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:02.463738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:02.463942 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:02.464094 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.464364 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.464536 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:37:02.467378 13355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:02.471019 13355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:02.479803 13486 tablet_service.cc:1467] Processing CreateTablet for tablet b692239dc5864f3b9e7a973128cedfcd (DEFAULT_TABLE table=test-table [id=f2acddcc8c9a4a1b81f4e9d37844ada6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:02.480551 13486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b692239dc5864f3b9e7a973128cedfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:02.483155 13441 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap starting.
I20260808 18:37:02.484017 13441 tablet_bootstrap.cc:654] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:02.484869 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.478780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:45615 (local address 127.0.0.1:52234)
0808 18:37:02.479026 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.479051 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.479140 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.479943 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.479947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.479966 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:02.480083 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.480090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.481760 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.481763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.483043 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.483049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.484288 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.484291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.484436 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.484529 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.484621 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.484704 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":9,"thread_start_us":125,"threads_started":1}
I20260808 18:37:02.484987 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.479091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.27.251.194:45615)
0808 18:37:02.479473 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.479485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.479509 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:37:02.479594 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.479599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.479777 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.480244 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:02.480414 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.481636 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.483207 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.484143 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.484298 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.484446 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.484676 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.484777 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":154,"threads_started":1}
I20260808 18:37:02.485577 13441 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: No bootstrap required, opened a new log
I20260808 18:37:02.485698 13441 ts_tablet_manager.cc:1397] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 18:37:02.486217 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.478287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:36557 (local address 127.0.0.1:37570)
0808 18:37:02.478455 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.478470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.478551 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.479625 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.479629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.479650 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:02.479775 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.479784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.481549 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.481555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.483035 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.483047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.485782 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.485786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.485908 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.485931 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.486017 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.486101 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260808 18:37:02.486195 13441 raft_consensus.cc:348] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.486311 13441 raft_consensus.cc:374] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:02.486326 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.478479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.27.251.195:36557)
0808 18:37:02.478990 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.479005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.479044 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:37:02.479139 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.479146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.479434 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.479651 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:02.479943 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.481417 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.483975 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.485590 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.485782 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.485923 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.486060 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.486162 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":168,"threads_started":1}
I20260808 18:37:02.486361 13441 raft_consensus.cc:729] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbf0b5f2c104c17ae7b08f768e674cf, State: Initialized, Role: FOLLOWER
I20260808 18:37:02.486503 13441 consensus_queue.cc:260] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.486769 13441 ts_tablet_manager.cc:1428] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 18:37:02.486663 13633 tablet_service.cc:1467] Processing CreateTablet for tablet b692239dc5864f3b9e7a973128cedfcd (DEFAULT_TABLE table=test-table [id=f2acddcc8c9a4a1b81f4e9d37844ada6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:02.487943 13633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b692239dc5864f3b9e7a973128cedfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:02.489475 13774 tablet_service.cc:1467] Processing CreateTablet for tablet b692239dc5864f3b9e7a973128cedfcd (DEFAULT_TABLE table=test-table [id=f2acddcc8c9a4a1b81f4e9d37844ada6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:02.490377 13774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b692239dc5864f3b9e7a973128cedfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:02.493374 13588 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Bootstrap starting.
I20260808 18:37:02.495653 13588 tablet_bootstrap.cc:654] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:02.496637 13588 log.cc:824] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:02.497454 13859 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Bootstrap starting.
I20260808 18:37:02.498766 13588 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: No bootstrap required, opened a new log
I20260808 18:37:02.498907 13588 ts_tablet_manager.cc:1397] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260808 18:37:02.499990 13859 tablet_bootstrap.cc:654] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:02.500835 13859 log.cc:824] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Log is configured to *not* fsync() on all Append() calls
W20260808 18:37:02.501765 13557 tablet.cc:2369] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:02.501883 13588 raft_consensus.cc:348] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.502137 13588 raft_consensus.cc:374] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:02.502202 13588 raft_consensus.cc:729] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f418cd91824eb1a0b8321ad7a5a317, State: Initialized, Role: FOLLOWER
I20260808 18:37:02.502722 13588 consensus_queue.cc:260] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.503070 13859 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: No bootstrap required, opened a new log
I20260808 18:37:02.503262 13859 ts_tablet_manager.cc:1397] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:37:02.503511 13702 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
I20260808 18:37:02.503770 13588 ts_tablet_manager.cc:1428] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260808 18:37:02.506803 13859 raft_consensus.cc:348] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.507130 13859 raft_consensus.cc:374] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:02.507205 13859 raft_consensus.cc:729] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbdfb52d582428a8c3e44886167ed1f, State: Initialized, Role: FOLLOWER
I20260808 18:37:02.507345 13563 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:02.507541 13563 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.507642 13859 consensus_queue.cc:260] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.508309 13846 heartbeater.cc:503] Master 127.27.251.254:36251 was elected leader, sending a full tablet report...
I20260808 18:37:02.508486 13859 ts_tablet_manager.cc:1428] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:37:02.508806 13563 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
I20260808 18:37:02.513561 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.509313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:35197 (local address 127.27.251.195:36557)
0808 18:37:02.509511 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.509516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.509529 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:37:02.509573 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.509576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.509619 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.509701 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:02.510090 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.510769 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.512373 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.512893 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.513048 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.513106 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.513346 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.513425 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260808 18:37:02.513561 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.509061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:36557 (local address 127.27.251.193:35197)
0808 18:37:02.509203 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.509222 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.509321 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.509741 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.509745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.509770 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:02.509932 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.509940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.510901 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.510906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.512234 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.512246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.513039 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.513043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.513166 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.513221 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.513326 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.513425 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260808 18:37:02.513686 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.509491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:46757 (local address 127.27.251.194:45615)
0808 18:37:02.509613 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.509617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.509890 (+ 273us) server_negotiation.cc:407] Connection header received
0808 18:37:02.510118 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.510121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.510170 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.510250 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:02.510991 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.511467 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.512594 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.513104 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.513239 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.513286 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.513488 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.513563 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:37:02.513715 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.509372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:45615 (local address 127.27.251.193:46757)
0808 18:37:02.509804 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.509818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:02.509952 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:02.510266 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:02.510269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:02.510286 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:02.510420 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.510428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.511595 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.511598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.512438 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:02.512448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.513242 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.513246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:02.513353 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.513372 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:37:02.513457 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 18:37:02.513531 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":80,"threads_started":1}
I20260808 18:37:02.514701 13795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692239dc5864f3b9e7a973128cedfcd" candidate_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbdfb52d582428a8c3e44886167ed1f" is_pre_election: true
I20260808 18:37:02.514683 13653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692239dc5864f3b9e7a973128cedfcd" candidate_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f418cd91824eb1a0b8321ad7a5a317" is_pre_election: true
I20260808 18:37:02.514969 13795 raft_consensus.cc:2454] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbbf0b5f2c104c17ae7b08f768e674cf in term 0.
I20260808 18:37:02.514987 13653 raft_consensus.cc:2454] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbbf0b5f2c104c17ae7b08f768e674cf in term 0.
I20260808 18:37:02.515496 13438 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbbdfb52d582428a8c3e44886167ed1f, dbbf0b5f2c104c17ae7b08f768e674cf; no voters:
I20260808 18:37:02.515802 13563 raft_consensus.cc:2781] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:02.515882 13563 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:02.515910 13563 raft_consensus.cc:3037] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:02.517118 13563 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.517452 13563 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 1 election: Requested vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
I20260808 18:37:02.517899 13795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692239dc5864f3b9e7a973128cedfcd" candidate_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbdfb52d582428a8c3e44886167ed1f"
I20260808 18:37:02.518031 13795 raft_consensus.cc:3037] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:02.518042 13653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b692239dc5864f3b9e7a973128cedfcd" candidate_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f418cd91824eb1a0b8321ad7a5a317"
I20260808 18:37:02.518205 13653 raft_consensus.cc:3037] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:02.519057 13795 raft_consensus.cc:2454] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbbf0b5f2c104c17ae7b08f768e674cf in term 1.
I20260808 18:37:02.519311 13653 raft_consensus.cc:2454] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbbf0b5f2c104c17ae7b08f768e674cf in term 1.
I20260808 18:37:02.519371 13438 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbbdfb52d582428a8c3e44886167ed1f, dbbf0b5f2c104c17ae7b08f768e674cf; no voters:
I20260808 18:37:02.519610 13563 raft_consensus.cc:2781] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:02.519873 13563 raft_consensus.cc:686] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 LEADER]: Becoming Leader. State: Replica: dbbf0b5f2c104c17ae7b08f768e674cf, State: Running, Role: LEADER
I20260808 18:37:02.520159 13563 consensus_queue.cc:237] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:02.522835 13355 catalog_manager.cc:5495] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf reported cstate change: term changed from 0 to 1, leader changed from <none> to dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193). New cstate: current_term: 1 leader_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } health_report { overall_health: HEALTHY } } }
I20260808 18:37:02.538296 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.534590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.27.251.193:37121)
0808 18:37:02.534725 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:02.534730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:02.534814 (+ 84us) server_negotiation.cc:407] Connection header received
0808 18:37:02.535001 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:02.535004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:02.535063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:02.535170 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:02.535416 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.535942 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.536851 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:02.537099 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:02.537218 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:02.537502 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:02.537907 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:02.538033 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:02.538103 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 18:37:02.538157 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:37:02.542214 13653 raft_consensus.cc:1261] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Refusing update from remote peer dbbf0b5f2c104c17ae7b08f768e674cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:02.542222 13795 raft_consensus.cc:1261] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Refusing update from remote peer dbbf0b5f2c104c17ae7b08f768e674cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:02.542912 13563 consensus_queue.cc:1035] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:02.543128 13865 consensus_queue.cc:1035] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:02.551097 13868 mvcc.cc:205] Tried to move back new op lower bound from 7316333455526162432 to 7316333455445315584. Current Snapshot: MvccSnapshot[applied={T|T < 7316333455526162432}]
I20260808 18:37:02.559700 13354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:02.559878 13354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:02.562270 13354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Sending DeleteTablet for 3 replicas of tablet b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:02.563011 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13395
I20260808 18:37:02.562947 13774 tablet_service.cc:1514] Processing DeleteTablet for tablet b692239dc5864f3b9e7a973128cedfcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:02 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:37570
I20260808 18:37:02.563081 13633 tablet_service.cc:1514] Processing DeleteTablet for tablet b692239dc5864f3b9e7a973128cedfcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:02 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:52234
I20260808 18:37:02.564381 13879 tablet_replica.cc:330] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: stopping tablet replica
I20260808 18:37:02.564955 13879 raft_consensus.cc:2229] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:02.565341 13879 raft_consensus.cc:2258] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:02.567256 13879 ts_tablet_manager.cc:1905] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:02.569628 13878 tablet_replica.cc:330] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: stopping tablet replica
I20260808 18:37:02.569819 13879 ts_tablet_manager.cc:1918] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:02.569908 13879 log.cc:1196] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:02.569983 13878 raft_consensus.cc:2229] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:02.570261 13879 ts_tablet_manager.cc:1939] T b692239dc5864f3b9e7a973128cedfcd P 10f418cd91824eb1a0b8321ad7a5a317: Deleting consensus metadata
I20260808 18:37:02.570317 13878 raft_consensus.cc:2258] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:02.571627 13342 catalog_manager.cc:4841] TS 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): tablet b692239dc5864f3b9e7a973128cedfcd (table test-table [id=f2acddcc8c9a4a1b81f4e9d37844ada6]) successfully deleted
I20260808 18:37:02.571823 13878 ts_tablet_manager.cc:1905] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 18:37:02.573640 13342 connection.cc:535] client connection to 127.27.251.193:37121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 18:37:02.573824 13342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:37:02.574141 13878 ts_tablet_manager.cc:1918] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:02.574242 13878 log.cc:1196] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:02.574453 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13560
I20260808 18:37:02.575446 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:02.574753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:37121 (local address 127.0.0.1:57752)
0808 18:37:02.575318 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:02.575378 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:37121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
I20260808 18:37:02.575786 13878 ts_tablet_manager.cc:1939] T b692239dc5864f3b9e7a973128cedfcd P bbbdfb52d582428a8c3e44886167ed1f: Deleting consensus metadata
W20260808 18:37:02.575642 13342 catalog_manager.cc:4584] TS dbbf0b5f2c104c17ae7b08f768e674cf (127.27.251.193:37121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b692239dc5864f3b9e7a973128cedfcd: Network error: Client connection negotiation failed: client connection to 127.27.251.193:37121: connect: Connection refused (error 111)
I20260808 18:37:02.577013 13342 catalog_manager.cc:4841] TS bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): tablet b692239dc5864f3b9e7a973128cedfcd (table test-table [id=f2acddcc8c9a4a1b81f4e9d37844ada6]) successfully deleted
I20260808 18:37:02.582086 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13705
I20260808 18:37:02.589361 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13321
I20260808 18:37:22.597596 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--webserver_interface=127.27.251.254
--webserver_port=38321
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:22.715977 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:22.716231 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:22.716271 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:22.720147 13881 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 18:37:22.720196 13881 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 18:37:22.720212 13881 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 18:37:22.720225 13881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:22.720243 13881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:22.720260 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:22.720276 13881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:22.720291 13881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:22.726207 13881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:36251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:22.727257 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:22.728277 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:37:22.737782 13881 server_base.cc:1034] running on GCE node
W20260808 18:37:22.737843 13890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:22.737766 13888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:22.737787 13887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:22.738548 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:22.739697 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:22.740899 13881 hybrid_clock.cc:648] HybridClock initialized: now 1786214242740867 us; error 44 us; skew 500 ppm
Aug 08 18:37:22 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:22.743646 13881 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:37:22.744577 13881 webserver.cc:460] Webserver started at http://127.27.251.254:38321/ using document root <none> and password file <none>
I20260808 18:37:22.745193 13881 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:22.745291 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:22.749185 13881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:37:22.752318 13897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:22.753582 13881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:22.753723 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "244b5b3c12f644708ad11aa3bd466313"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "dbe847a6270fa1ccb59b791c5f0f7981"
server_key_iv: "63911ebc4393ddf44f48c111c22007ad"
server_key_version: "encryptionkey@0"
I20260808 18:37:22.754295 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:22.767446 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:22.770337 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:22.770546 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:22.778002 13881 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:36251
I20260808 18:37:22.778307 13952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:36251 every 8 connection(s)
I20260808 18:37:22.779138 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:22.783449 13953 sys_catalog.cc:263] Verifying existing consensus state
I20260808 18:37:22.785117 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap starting.
I20260808 18:37:22.786233 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13881
I20260808 18:37:22.786692 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:37121
--local_ip_for_outbound_sockets=127.27.251.193
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=41249
--webserver_interface=127.27.251.193
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:22.810707 13953 log.cc:824] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:22.825874 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:37:22.826354 13953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Bootstrap complete.
I20260808 18:37:22.832204 13953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:22.832876 13953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Initialized, Role: FOLLOWER
I20260808 18:37:22.833488 13953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:22.833647 13953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:22.833719 13953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:22.833824 13953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 2 FOLLOWER]: Advancing to term 3
I20260808 18:37:22.836486 13953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:22.836890 13953 leader_election.cc:302] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244b5b3c12f644708ad11aa3bd466313; no voters:
I20260808 18:37:22.837260 13953 leader_election.cc:288] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 18:37:22.837415 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 18:37:22.837714 13957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [term 3 LEADER]: Becoming Leader. State: Replica: 244b5b3c12f644708ad11aa3bd466313, State: Running, Role: LEADER
I20260808 18:37:22.838038 13957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } }
I20260808 18:37:22.838640 13953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:22.839723 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244b5b3c12f644708ad11aa3bd466313. Latest consensus state: current_term: 3 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:37:22.839862 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:22.839709 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "244b5b3c12f644708ad11aa3bd466313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244b5b3c12f644708ad11aa3bd466313" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 36251 } } }
I20260808 18:37:22.839964 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:22.840278 13966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:22.845889 13966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:22.846741 13966 catalog_manager.cc:1259] Loaded cluster ID: bb66fe4f43bc4d6185a7ece591a83f7a
I20260808 18:37:22.846804 13966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:22.849433 13966 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:22.850195 13966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Loaded TSK: 0
I20260808 18:37:22.850281 13966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 244b5b3c12f644708ad11aa3bd466313: Loaded TSK: 1
F20260808 18:37:22.851074 13966 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f471c7549ad google::LogMessage::Fail() at ??:0
@ 0x7f471c75788a google::LogMessage::SendToLog() at ??:0
@ 0x7f471c7544de google::LogMessage::Flush() at ??:0
@ 0x7f471c755a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f471f41818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f471f410f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f47270b97bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f472709399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4727094178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f47270d5df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f47270f482b std::function<>::operator()() at ??:0
@ 0x7f4727093b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4727094f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4727091579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f47270d5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f471e9a63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f471e9a6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f471e9a83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f471e997512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f471ffdb6db start_thread at ??:0
@ 0x7f471b25261f clone at ??:0
W20260808 18:37:22.943439 13955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:22.943681 13955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:22.943724 13955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:22.948144 13955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:22.948194 13955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:22.948254 13955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:22.954640 13955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=41249
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:22.955879 13955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:22.957031 13955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:22.967531 14004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:22.967689 14007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:22.967526 14005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:22.968021 13955 server_base.cc:1034] running on GCE node
I20260808 18:37:22.968493 13955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:22.969431 13955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:22.970642 13955 hybrid_clock.cc:648] HybridClock initialized: now 1786214242970602 us; error 52 us; skew 500 ppm
Aug 08 18:37:22 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:22.974198 13955 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:37:22.975101 13955 webserver.cc:460] Webserver started at http://127.27.251.193:41249/ using document root <none> and password file <none>
I20260808 18:37:22.975744 13955 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:22.975811 13955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:22.980183 13955 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:37:22.982584 14014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:22.983906 13955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 18:37:22.984045 13955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "dbbf0b5f2c104c17ae7b08f768e674cf"
format_stamp: "Formatted at 2026-08-08 18:36:40 on dist-test-slave-fd7l"
server_key: "86b00ddf0f3a0be120184d0459223d45"
server_key_iv: "46e0e3bba105699a8e1354810fb6b5d5"
server_key_version: "encryptionkey@0"
I20260808 18:37:22.984676 13955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:23.002314 13955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:23.005884 13955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:23.006107 13955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:23.006767 13955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:23.008004 14021 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 18:37:23.013748 13955 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 18:37:23.013831 13955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 18:37:23.013922 13955 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 18:37:23.015609 13955 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 18:37:23.015679 13955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 18:37:23.015996 14021 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap starting.
I20260808 18:37:23.027032 13955 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:37121
I20260808 18:37:23.027367 14133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:37121 every 8 connection(s)
I20260808 18:37:23.028119 13955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:23.037453 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 13955
I20260808 18:37:23.037926 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:45615
--local_ip_for_outbound_sockets=127.27.251.194
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=43417
--webserver_interface=127.27.251.194
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:23.039043 14021 log.cc:824] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:23.046430 14021 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 18:37:23.046840 14021 tablet_bootstrap.cc:492] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Bootstrap complete.
I20260808 18:37:23.048096 14021 ts_tablet_manager.cc:1397] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260808 18:37:23.050594 14021 raft_consensus.cc:348] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:23.051023 14021 raft_consensus.cc:729] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbf0b5f2c104c17ae7b08f768e674cf, State: Initialized, Role: FOLLOWER
I20260808 18:37:23.051509 14021 consensus_queue.cc:260] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:23.052454 14021 ts_tablet_manager.cc:1428] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260808 18:37:23.136327 14137 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:37:23.030223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:35941)
0808 18:37:23.030738 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.030795 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:23.031657 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:23.135530 (+103873us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":144,"threads_started":1}
W20260808 18:37:23.137008 14134 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36251 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:36251: Client connection negotiation failed: client connection to 127.27.251.254:36251: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 18:37:23.137764 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.137489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:45011)
0808 18:37:23.137622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.137695 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 18:37:23.138556 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.138301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:55591)
0808 18:37:23.138429 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.138495 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:37:23.138729 14134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:37:23.183395 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:23.183651 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:23.183696 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:23.187732 14140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:23.187789 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:23.187851 14140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:23.193078 14140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:45615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=43417
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:23.194089 14140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:23.195086 14140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:23.205240 14147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:23.205240 14149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:23.205226 14146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:23.205289 14140 server_base.cc:1034] running on GCE node
I20260808 18:37:23.205785 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:23.206491 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:23.207676 14140 hybrid_clock.cc:648] HybridClock initialized: now 1786214243207643 us; error 46 us; skew 500 ppm
Aug 08 18:37:23 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:23.210446 14140 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:37:23.211336 14140 webserver.cc:460] Webserver started at http://127.27.251.194:43417/ using document root <none> and password file <none>
I20260808 18:37:23.211892 14140 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:23.211987 14140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:23.215734 14140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:37:23.217810 14157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.218943 14140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:23.219076 14140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "10f418cd91824eb1a0b8321ad7a5a317"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "1977edc1be4abc8840a569785d3fc4a1"
server_key_iv: "fec9d58e9f45a6639da3cbcb4152a894"
server_key_version: "encryptionkey@0"
I20260808 18:37:23.219635 14140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:23.236061 14140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:23.239117 14140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:23.239337 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:23.239984 14140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:23.240886 14140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:23.240962 14140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.241039 14140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:23.241091 14140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.251078 14140 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:45615
I20260808 18:37:23.251142 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:45615 every 8 connection(s)
I20260808 18:37:23.252580 14140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:23.256376 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.254781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:50627)
0808 18:37:23.255495 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.255609 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":109,"threads_started":1}
W20260808 18:37:23.256942 14279 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36251 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:36251: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
I20260808 18:37:23.257217 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14140
I20260808 18:37:23.257743 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.257396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:55651)
0808 18:37:23.257596 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.257665 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 18:37:23.257771 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:36557
--local_ip_for_outbound_sockets=127.27.251.195
--tserver_master_addrs=127.27.251.254:36251
--webserver_port=45087
--webserver_interface=127.27.251.195
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:37:23.258507 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.258308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:35555)
0808 18:37:23.258415 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.258460 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:37:23.258771 14279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 18:37:23.373207 14282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:23.373504 14282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:23.373595 14282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:23.377646 14282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:23.377727 14282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:23.377811 14282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:23.383117 14282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:36557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=45087
--tserver_master_addrs=127.27.251.254:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:23.384207 14282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:23.385311 14282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:23.395795 14292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:23.395854 14290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:23.395877 14288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:23.396116 14282 server_base.cc:1034] running on GCE node
I20260808 18:37:23.396718 14282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:23.397424 14282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:23.398665 14282 hybrid_clock.cc:648] HybridClock initialized: now 1786214243398638 us; error 47 us; skew 500 ppm
Aug 08 18:37:23 dist-test-slave-fd7l krb5kdc[12745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:23.401820 14282 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:37:23.402747 14282 webserver.cc:460] Webserver started at http://127.27.251.195:45087/ using document root <none> and password file <none>
I20260808 18:37:23.403340 14282 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:23.403437 14282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:23.407394 14282 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:37:23.409530 14299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.410742 14282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:23.410879 14282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "bbbdfb52d582428a8c3e44886167ed1f"
format_stamp: "Formatted at 2026-08-08 18:36:41 on dist-test-slave-fd7l"
server_key: "8e76066c7d0395eab892d02d014b8b28"
server_key_iv: "02b0da26b3f5513fb13ef963d5df2525"
server_key_version: "encryptionkey@0"
I20260808 18:37:23.411434 14282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:23.420538 14282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:23.423564 14282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:23.423770 14282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:23.424343 14282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:23.425177 14282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:23.425249 14282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.425320 14282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:23.425364 14282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:23.434154 14282 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:36557
I20260808 18:37:23.434202 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:36557 every 8 connection(s)
I20260808 18:37:23.435149 14282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:23.435482 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14282
I20260808 18:37:23.438923 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.437467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:35589)
0808 18:37:23.438133 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.438232 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":140,"threads_started":1}
W20260808 18:37:23.439347 14422 heartbeater.cc:644] Failed to heartbeat to 127.27.251.254:36251 (0 consecutive failures): Network error: Failed to ping master at 127.27.251.254:36251: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
I20260808 18:37:23.439886 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.439693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:60841)
0808 18:37:23.439791 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.439837 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 18:37:23.440562 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:23.440343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:39829)
0808 18:37:23.440432 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:23.440471 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 18:37:23.440780 14422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 18:37:24.139812 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.139375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:49173)
0808 18:37:24.139631 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:24.139696 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260808 18:37:24.259929 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.259488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:56565)
0808 18:37:24.259759 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:24.259823 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260808 18:37:24.441707 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.441306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:49363)
0808 18:37:24.441544 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:24.441599 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260808 18:37:24.912981 14430 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:24.913232 14430 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:24.914345 14430 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
Aug 08 18:37:24 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.195@KRBTEST.COM
Aug 08 18:37:24 dist-test-slave-fd7l krb5kdc[12745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.194@KRBTEST.COM
I20260808 18:37:24.935607 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.914864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:45707 (local address 127.27.251.195:36557)
0808 18:37:24.915450 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:24.915482 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:24.915534 (+ 52us) server_negotiation.cc:407] Connection header received
0808 18:37:24.916557 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:24.916566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:24.917028 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:24.917391 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:24.919464 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.921105 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.923379 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.924127 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.924303 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:24.930819 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:24.930826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:24.930845 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:24.930878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:24.932885 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933487 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:24.933497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:24.933504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:24.933640 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:24.934024 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:24.934027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:24.934029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:24.934443 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:24.934585 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:24.935159 (+ 574us) server_negotiation.cc:299] Negotiation successful
0808 18:37:24.935428 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":62,"threads_started":1}
I20260808 18:37:24.935622 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.915400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:57873 (local address 127.27.251.194:45615)
0808 18:37:24.915780 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:24.915793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:24.915828 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:37:24.916667 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:24.916674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:24.917028 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:24.917391 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:24.919469 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.921105 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.923373 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.924127 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.924303 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:24.931106 (+ 6803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:24.931112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:24.931125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:24.931154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:24.933008 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933547 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:24.933552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:24.933556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:24.933673 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:24.934023 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:24.934027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:24.934029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:24.934444 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:24.934584 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:24.935140 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 18:37:24.935440 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260808 18:37:24.935606 14434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.915140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:45615 (local address 127.27.251.193:57873)
0808 18:37:24.915475 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:24.915504 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:24.915616 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:24.917510 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:24.917517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:24.918233 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:24.919244 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:24.919268 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.921304 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.921308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:24.923206 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:24.923219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.924304 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.924309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:24.924658 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:24.928684 (+ 4026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:24.928712 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:24.930867 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:24.933144 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:24.933148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:24.933424 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:24.933809 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:24.933812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:24.933918 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:24.934585 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:24.934588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:24.934829 (+ 241us) client_negotiation.cc:769] Sending connection context
0808 18:37:24.935021 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 18:37:24.935389 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":388,"spinlock_wait_cycles":48256,"thread_start_us":67,"threads_started":1}
I20260808 18:37:24.935731 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:24.914739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:36557 (local address 127.27.251.193:45707)
0808 18:37:24.915269 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:24.915286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:24.915408 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:24.917510 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:24.917517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:24.918233 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:24.919244 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:24.919268 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.921304 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.921308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:24.923206 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:24.923219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:24.924304 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:24.924309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:24.924657 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:24.928657 (+ 4000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:24.928688 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:24.930544 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:24.933008 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:24.933028 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:24.933367 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:24.933758 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:24.933762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:24.933764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:24.933918 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:24.934576 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:24.934582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:24.934826 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 18:37:24.935021 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 18:37:24.935531 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":689,"spinlock_wait_cycles":10368,"thread_start_us":89,"threads_started":1}
W20260808 18:37:24.938900 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
W20260808 18:37:24.939123 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:24.939172 14018 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbbf0b5f2c104c17ae7b08f768e674cf; no voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f
I20260808 18:37:24.939421 14430 raft_consensus.cc:2726] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:37:25.141202 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:25.140836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:41821)
0808 18:37:25.141002 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:25.141094 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 18:37:25.261210 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:25.260781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:58699)
0808 18:37:25.261037 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:25.261095 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260808 18:37:25.442854 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:25.442414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:39553)
0808 18:37:25.442712 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:25.442764 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":137,"threads_started":1}
I20260808 18:37:26.142495 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:26.142040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:37127)
0808 18:37:26.142336 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:26.142395 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260808 18:37:26.262458 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:26.261958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:45315)
0808 18:37:26.262290 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:26.262356 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":87,"threads_started":1}
I20260808 18:37:26.444036 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:26.443584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:47081)
0808 18:37:26.443865 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:26.443922 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260808 18:37:27.051079 14442 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:27.051263 14442 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:27.051760 14442 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
W20260808 18:37:27.052690 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
W20260808 18:37:27.052806 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:27.052889 14018 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbbf0b5f2c104c17ae7b08f768e674cf; no voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f
I20260808 18:37:27.053103 14442 raft_consensus.cc:2726] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:37:27.143846 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:27.143373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:51269)
0808 18:37:27.143682 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:27.143741 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260808 18:37:27.263805 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:27.263303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:56177)
0808 18:37:27.263626 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:27.263683 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260808 18:37:27.445274 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:27.444843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:33257)
0808 18:37:27.445102 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:27.445175 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260808 18:37:28.145393 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:28.144910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:60233)
0808 18:37:28.145235 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:28.145296 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260808 18:37:28.265189 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:28.264744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:55799)
0808 18:37:28.265024 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:28.265087 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260808 18:37:28.446503 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:28.446035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:41275)
0808 18:37:28.446326 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:28.446388 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260808 18:37:29.146715 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:29.146244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:38935)
0808 18:37:29.146537 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:29.146598 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260808 18:37:29.266551 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:29.266039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:52183)
0808 18:37:29.266346 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:29.266408 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260808 18:37:29.447816 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:29.447350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:36921)
0808 18:37:29.447615 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:29.447675 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260808 18:37:30.037818 14453 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:30.037946 14453 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:30.038398 14453 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
W20260808 18:37:30.039300 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
W20260808 18:37:30.039441 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:30.039521 14018 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbbf0b5f2c104c17ae7b08f768e674cf; no voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f
I20260808 18:37:30.039729 14453 raft_consensus.cc:2726] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:37:30.147921 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:30.147496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:44213)
0808 18:37:30.147760 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:30.147819 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260808 18:37:30.267844 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:30.267340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:34591)
0808 18:37:30.267685 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:30.267744 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
I20260808 18:37:30.449241 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:30.448739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:49833)
0808 18:37:30.449073 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:30.449138 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":75,"threads_started":1}
I20260808 18:37:31.149226 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:31.148784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:45917)
0808 18:37:31.149063 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:31.149126 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260808 18:37:31.269049 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:31.268631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:39555)
0808 18:37:31.268879 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:31.268950 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260808 18:37:31.450474 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:31.450009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:45045)
0808 18:37:31.450295 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:31.450373 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260808 18:37:32.150434 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:32.150020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:46759)
0808 18:37:32.150281 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:32.150339 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260808 18:37:32.270362 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:32.269864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:44491)
0808 18:37:32.270197 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:32.270262 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260808 18:37:32.451753 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:32.451300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:59149)
0808 18:37:32.451598 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:32.451656 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260808 18:37:33.151893 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:33.151436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:57449)
0808 18:37:33.151737 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:33.151797 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260808 18:37:33.271720 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:33.271193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:56659)
0808 18:37:33.271533 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:33.271594 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260808 18:37:33.453578 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:33.452873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:33961)
0808 18:37:33.453286 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:33.453416 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":165,"threads_started":1}
I20260808 18:37:33.662544 14466 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:33.662750 14466 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:33.663300 14466 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
W20260808 18:37:33.664598 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
W20260808 18:37:33.664844 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:33.664914 14018 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbbf0b5f2c104c17ae7b08f768e674cf; no voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f
I20260808 18:37:33.665104 14466 raft_consensus.cc:2726] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:37:34.153810 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:34.153088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:34213)
0808 18:37:34.153512 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:34.153617 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":109,"threads_started":1}
I20260808 18:37:34.273443 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:34.272745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:47461)
0808 18:37:34.273178 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:34.273277 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":160,"threads_started":1}
I20260808 18:37:34.455286 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:34.454633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:54847)
0808 18:37:34.455041 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:34.455136 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260808 18:37:35.155289 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:35.154833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:38299)
0808 18:37:35.155118 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:35.155184 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260808 18:37:35.275009 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:35.274395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:45133)
0808 18:37:35.274771 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:35.274855 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":146,"threads_started":1}
I20260808 18:37:35.456749 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:35.456125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:36467)
0808 18:37:35.456414 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:35.456472 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260808 18:37:35.908612 14473 raft_consensus.cc:482] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:35.908766 14473 raft_consensus.cc:504] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbbdfb52d582428a8c3e44886167ed1f" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 36557 } } peers { permanent_uuid: "10f418cd91824eb1a0b8321ad7a5a317" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 45615 } } peers { permanent_uuid: "dbbf0b5f2c104c17ae7b08f768e674cf" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 37121 } }
I20260808 18:37:35.909243 14473 leader_election.cc:288] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557), 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615)
W20260808 18:37:35.910183 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10f418cd91824eb1a0b8321ad7a5a317 (127.27.251.194:45615): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
W20260808 18:37:35.910311 14018 leader_election.cc:341] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbbdfb52d582428a8c3e44886167ed1f (127.27.251.195:36557): Not found: Tablet not found: b692239dc5864f3b9e7a973128cedfcd
I20260808 18:37:35.910341 14018 leader_election.cc:302] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbbf0b5f2c104c17ae7b08f768e674cf; no voters: 10f418cd91824eb1a0b8321ad7a5a317, bbbdfb52d582428a8c3e44886167ed1f
I20260808 18:37:35.910524 14473 raft_consensus.cc:2726] T b692239dc5864f3b9e7a973128cedfcd P dbbf0b5f2c104c17ae7b08f768e674cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 18:37:36.156694 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:36.156143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:36799)
0808 18:37:36.156460 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:36.156582 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260808 18:37:36.276328 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:36.275880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:42959)
0808 18:37:36.276162 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:36.276225 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260808 18:37:36.457929 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:36.457517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:60943)
0808 18:37:36.457778 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:36.457832 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260808 18:37:37.157987 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:37.157481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:49721)
0808 18:37:37.157810 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:37.157876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
I20260808 18:37:37.277700 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:37.277237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:49689)
0808 18:37:37.277520 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:37.277587 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260808 18:37:37.459095 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:37.458681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.195:42401)
0808 18:37:37.458943 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:37.458996 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260808 18:37:38.159497 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:38.159038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.193:57103)
0808 18:37:38.159335 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:38.159397 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260808 18:37:38.279039 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:38.278606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:36251 (local address 127.27.251.194:57847)
0808 18:37:38.278890 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:38.278944 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.254:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260808 18:37:38.439327 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 13955
I20260808 18:37:38.450534 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14140
I20260808 18:37:38.458104 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14282
2026-08-08T18:37:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59215 ms)
[----------] 30 tests from SecurityITest (251062 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:38 dist-test-slave-fd7l krb5kdc[14484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:38 dist-test-slave-fd7l krb5kdc[14484](info): set up 2 sockets
Aug 08 18:37:38 dist-test-slave-fd7l krb5kdc[14484](info): commencing operation
krb5kdc: starting...
W20260808 18:37:39.575125 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:39 dist-test-slave-fd7l krb5kdc[14484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:37:39.705667 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34611
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:39.822626 14498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:39.823092 14498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:39.823236 14498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:39.827418 14498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:39.827519 14498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:39.827567 14498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:39.827608 14498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:39.827643 14498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:39.833186 14498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:39.834337 14498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:39.835470 14498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:39.844676 14503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:39.844724 14498 server_base.cc:1034] running on GCE node
W20260808 18:37:39.844595 14504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:39.844682 14506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:39.845369 14498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:39.846516 14498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:39.847728 14498 hybrid_clock.cc:648] HybridClock initialized: now 1786214259847696 us; error 42 us; skew 500 ppm
Aug 08 18:37:39 dist-test-slave-fd7l krb5kdc[14484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:39.850728 14498 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:37:39.851686 14498 webserver.cc:460] Webserver started at http://127.27.251.254:46227/ using document root <none> and password file <none>
I20260808 18:37:39.852244 14498 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:39.852310 14498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:39.852557 14498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:39.854699 14498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "b8eb30785c0d45ee99d0f0bb3a714747"
format_stamp: "Formatted at 2026-08-08 18:37:39 on dist-test-slave-fd7l"
server_key: "0b3e686c124cb1c398fd9c9f31ff3634"
server_key_iv: "ba6f9f21ea029f30e88d3524c7c2d364"
server_key_version: "encryptionkey@0"
I20260808 18:37:39.855298 14498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "b8eb30785c0d45ee99d0f0bb3a714747"
format_stamp: "Formatted at 2026-08-08 18:37:39 on dist-test-slave-fd7l"
server_key: "0b3e686c124cb1c398fd9c9f31ff3634"
server_key_iv: "ba6f9f21ea029f30e88d3524c7c2d364"
server_key_version: "encryptionkey@0"
I20260808 18:37:39.859400 14498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:37:39.862210 14513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:39.863593 14498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 18:37:39.863703 14498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "b8eb30785c0d45ee99d0f0bb3a714747"
format_stamp: "Formatted at 2026-08-08 18:37:39 on dist-test-slave-fd7l"
server_key: "0b3e686c124cb1c398fd9c9f31ff3634"
server_key_iv: "ba6f9f21ea029f30e88d3524c7c2d364"
server_key_version: "encryptionkey@0"
I20260808 18:37:39.863799 14498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:39.876981 14498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:39.880158 14498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:39.880328 14498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:39.888422 14498 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:34611
I20260808 18:37:39.888425 14572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:34611 every 8 connection(s)
I20260808 18:37:39.889564 14498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:39.892542 14573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:39.892575 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14498
I20260808 18:37:39.892752 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:39.893100 28655 external_mini_cluster.cc:1428] Setting key 2114424638669be9b2d7b6b51bd51c1e
I20260808 18:37:39.899570 14573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: Bootstrap starting.
I20260808 18:37:39.902439 14573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: Neither blocks nor log segments found. Creating new log.
Aug 08 18:37:39 dist-test-slave-fd7l krb5kdc[14484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:39.903296 14573 log.cc:824] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:39.905437 14573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: No bootstrap required, opened a new log
I20260808 18:37:39.908476 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:39.894567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.27.251.254:34611)
0808 18:37:39.895022 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:39.895042 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:39.895082 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:37:39.895816 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:39.895834 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:39.896216 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:39.896578 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:39.897512 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:39.898611 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:39.899760 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:39.900277 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:39.900412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:39.903556 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:39.903562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:39.903580 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:39.903611 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:39.905431 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:39.906029 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:39.906034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:39.906039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:39.906149 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:39.906455 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:39.906457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:39.906458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:39.906837 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:39.906998 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:39.907333 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 18:37:39.907705 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":142,"threads_started":1}
I20260808 18:37:39.909758 14573 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } }
I20260808 18:37:39.910038 14573 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:39.910120 14573 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8eb30785c0d45ee99d0f0bb3a714747, State: Initialized, Role: FOLLOWER
I20260808 18:37:39.910580 14573 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } }
I20260808 18:37:39.910772 14573 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:39.910884 14573 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:39.911011 14573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:39.911955 14573 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } }
I20260808 18:37:39.912310 14573 leader_election.cc:302] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8eb30785c0d45ee99d0f0bb3a714747; no voters:
I20260808 18:37:39.912685 14573 leader_election.cc:288] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:39.912815 14578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:39.913054 14578 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [term 1 LEADER]: Becoming Leader. State: Replica: b8eb30785c0d45ee99d0f0bb3a714747, State: Running, Role: LEADER
I20260808 18:37:39.913962 14578 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } }
I20260808 18:37:39.914017 14573 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:39.915681 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8eb30785c0d45ee99d0f0bb3a714747. Latest consensus state: current_term: 1 leader_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } } }
I20260808 18:37:39.915709 14579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8eb30785c0d45ee99d0f0bb3a714747" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34611 } } }
I20260808 18:37:39.915800 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:39.915827 14579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:39.916237 14596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:39.919420 14596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:39.925282 14596 catalog_manager.cc:1347] Generated new cluster ID: 3a2495b400d14a7cb27bbe037a0a5298
I20260808 18:37:39.925360 14596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:39.937145 14596 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:39.938159 14596 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:39.947185 14596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: Generated new TSK 0
I20260808 18:37:39.947907 14596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:37:39.984228 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:40.099562 14603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:40.099871 14603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:40.099951 14603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:40.104058 14603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:40.104147 14603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:40.104239 14603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:40.109658 14603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:40.110802 14603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:40.111864 14603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:40.122314 14608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:40.122329 14609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:40.122372 14611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:40.122416 14603 server_base.cc:1034] running on GCE node
I20260808 18:37:40.122998 14603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:40.123675 14603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:40.124862 14603 hybrid_clock.cc:648] HybridClock initialized: now 1786214260124841 us; error 33 us; skew 500 ppm
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:40.127712 14603 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:37:40.128666 14603 webserver.cc:460] Webserver started at http://127.27.251.193:37229/ using document root <none> and password file <none>
I20260808 18:37:40.129245 14603 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:40.129338 14603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:40.129578 14603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:40.131738 14603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "64762e84694b49a794059f70f159dc8e"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "73960812243c8ffd2b39bf74219293c8"
server_key_iv: "ce4f00f7cc2ee247f7be9c64ea680bb6"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.132418 14603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "64762e84694b49a794059f70f159dc8e"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "73960812243c8ffd2b39bf74219293c8"
server_key_iv: "ce4f00f7cc2ee247f7be9c64ea680bb6"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.136291 14603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 18:37:40.138944 14618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.140264 14603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 18:37:40.140359 14603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "64762e84694b49a794059f70f159dc8e"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "73960812243c8ffd2b39bf74219293c8"
server_key_iv: "ce4f00f7cc2ee247f7be9c64ea680bb6"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.140432 14603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:40.162997 14603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:40.166581 14603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:40.166836 14603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:40.167537 14603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:40.168571 14603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:40.168630 14603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.168695 14603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:40.168726 14603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.178447 14603 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:44505
I20260808 18:37:40.178469 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:44505 every 8 connection(s)
I20260808 18:37:40.179396 14603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:40.181630 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14603
I20260808 18:37:40.181766 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:40.182103 28655 external_mini_cluster.cc:1428] Setting key 59bc22380e16a5d70113955e0bb8b9e2
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:37:40.195712 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.181438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:49689 (local address 127.27.251.254:34611)
0808 18:37:40.181628 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.181632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.182743 (+ 1111us) server_negotiation.cc:407] Connection header received
0808 18:37:40.184408 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.184412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.184464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.184579 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:40.185804 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.186416 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.188038 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.188473 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.188633 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.191390 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:40.191397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:40.191403 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:40.191437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:40.193428 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.194039 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.194045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.194049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.194156 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.194484 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.194486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.194487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.194683 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:40.194770 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.195448 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.195580 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:37:40.196931 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.181950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34611 (local address 127.27.251.193:49689)
0808 18:37:40.182542 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.182608 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.184189 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.184757 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.184764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.185126 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:40.185645 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.185660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.186588 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.186594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.187902 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.187912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.188704 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.188707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.188871 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.189569 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:40.189587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:40.191157 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:40.193589 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.193594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.193608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.193941 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.194258 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.194260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.194261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.194402 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.194803 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:40.194809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:40.195215 (+ 406us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.195385 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.195669 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":174,"threads_started":1}
I20260808 18:37:40.198279 14738 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34611
I20260808 18:37:40.198544 14738 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:40.199081 14738 heartbeater.cc:511] Master 127.27.251.254:34611 requested a full tablet report, sending...
I20260808 18:37:40.200816 14532 ts_manager.cc:194] Registered new tserver with Master: 64762e84694b49a794059f70f159dc8e (127.27.251.193:44505)
I20260808 18:37:40.201978 14532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:49689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:37:40.216149 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:40.333794 14743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:40.334048 14743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:40.334090 14743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:40.337991 14743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:40.338042 14743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:40.338097 14743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:40.343427 14743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:40.344455 14743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:40.345521 14743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:37:40.355499 14743 server_base.cc:1034] running on GCE node
W20260808 18:37:40.355546 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:40.355414 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:40.355450 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:40.356120 14743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:40.356842 14743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:40.358001 14743 hybrid_clock.cc:648] HybridClock initialized: now 1786214260357984 us; error 30 us; skew 500 ppm
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:40.360821 14743 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:37:40.361831 14743 webserver.cc:460] Webserver started at http://127.27.251.194:39909/ using document root <none> and password file <none>
I20260808 18:37:40.362437 14743 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:40.362530 14743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:40.362772 14743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:40.364948 14743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "3c09d785b0804b9090dc005ac9540e3b"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "06db9bb59e41da99db311b0ec07f8a6d"
server_key_iv: "81bfa1a5299870fe38511cc9fb150048"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.365540 14743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "3c09d785b0804b9090dc005ac9540e3b"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "06db9bb59e41da99db311b0ec07f8a6d"
server_key_iv: "81bfa1a5299870fe38511cc9fb150048"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.369334 14743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:40.371742 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.373061 14743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:40.373202 14743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "3c09d785b0804b9090dc005ac9540e3b"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "06db9bb59e41da99db311b0ec07f8a6d"
server_key_iv: "81bfa1a5299870fe38511cc9fb150048"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.373318 14743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:40.385123 14743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:40.387907 14743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:40.388108 14743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:40.388751 14743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:40.389561 14743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:40.389652 14743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.389748 14743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:40.389802 14743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.398931 14743 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:36797
I20260808 18:37:40.398969 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:36797 every 8 connection(s)
I20260808 18:37:40.399822 14743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:40.404601 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14743
I20260808 18:37:40.404763 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:40.405066 28655 external_mini_cluster.cc:1428] Setting key 2cf1b19fb46bf0b3f11b3124ea55a047
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:40.415395 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.401703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:54443 (local address 127.27.251.254:34611)
0808 18:37:40.401871 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.401876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.402697 (+ 821us) server_negotiation.cc:407] Connection header received
0808 18:37:40.404003 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.404006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.404053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.404151 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:40.405525 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.406298 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.407476 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.407907 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.408031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.411006 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:40.411010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:40.411012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:40.411046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:40.412869 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.413444 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.413448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.413450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.413543 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.413912 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.413917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.413919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.414199 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:40.414327 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.415000 (+ 673us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.415152 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":66}
I20260808 18:37:40.416527 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.402077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34611 (local address 127.27.251.194:54443)
0808 18:37:40.402542 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.402585 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.403804 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.404325 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.404332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.404807 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:40.405376 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.405390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.406431 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.406434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.407335 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.407345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.408065 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.408071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.408294 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.408967 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:40.408995 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:40.410832 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:40.413018 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.413025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.413040 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.413318 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.413647 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.413651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.413654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.413806 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.414325 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:40.414332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:40.414733 (+ 401us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.414945 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.415354 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:37:40.417881 14881 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34611
I20260808 18:37:40.418171 14881 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:40.418650 14881 heartbeater.cc:511] Master 127.27.251.254:34611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:37:40.419857 14532 ts_manager.cc:194] Registered new tserver with Master: 3c09d785b0804b9090dc005ac9540e3b (127.27.251.194:36797)
I20260808 18:37:40.420495 14532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:54443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:37:40.435585 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:40.560863 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:40.561141 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:40.561221 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:40.565385 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:40.565454 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:40.565513 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:40.571010 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:40.572166 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:40.573241 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:40.583597 14894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:40.583649 14887 server_base.cc:1034] running on GCE node
W20260808 18:37:40.583662 14895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:40.583597 14897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:40.584296 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:40.585047 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:40.586248 14887 hybrid_clock.cc:648] HybridClock initialized: now 1786214260586228 us; error 32 us; skew 500 ppm
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:40.589514 14887 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:37:40.590481 14887 webserver.cc:460] Webserver started at http://127.27.251.195:45229/ using document root <none> and password file <none>
I20260808 18:37:40.591100 14887 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:40.591172 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:40.591393 14887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:40.593806 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "d2d4d844ade5419c96f97d0eecbf582a"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "a954299cbb5d675ad72e1777d6ea11e4"
server_key_iv: "78d13cf0c0b8252cfde19fad96550477"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.594373 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "d2d4d844ade5419c96f97d0eecbf582a"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "a954299cbb5d675ad72e1777d6ea11e4"
server_key_iv: "78d13cf0c0b8252cfde19fad96550477"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.598650 14887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:37:40.601311 14904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.602715 14887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:40.602819 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "d2d4d844ade5419c96f97d0eecbf582a"
format_stamp: "Formatted at 2026-08-08 18:37:40 on dist-test-slave-fd7l"
server_key: "a954299cbb5d675ad72e1777d6ea11e4"
server_key_iv: "78d13cf0c0b8252cfde19fad96550477"
server_key_version: "encryptionkey@0"
I20260808 18:37:40.602912 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:40.614972 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:40.618340 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:40.618513 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:40.619076 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:40.619890 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:40.619938 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.619980 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:40.619997 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:40.629828 14887 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:34267
I20260808 18:37:40.629848 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:34267 every 8 connection(s)
I20260808 18:37:40.630746 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:40.632642 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 14887
I20260808 18:37:40.632797 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:40.633116 28655 external_mini_cluster.cc:1428] Setting key 837e03b691774d70fd043d5dfcc03bce
Aug 08 18:37:40 dist-test-slave-fd7l krb5kdc[14484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214260, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:40.644980 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.632719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:51085 (local address 127.27.251.254:34611)
0808 18:37:40.632900 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.632905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.633877 (+ 972us) server_negotiation.cc:407] Connection header received
0808 18:37:40.635636 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.635639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.635683 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.635746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:40.637010 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.637545 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.638787 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.638994 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.639071 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.641632 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:40.641635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:40.641637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:40.641663 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:40.643009 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.643564 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.643567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.643568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.643648 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.643952 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.643954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.643955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.644129 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:40.644234 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.644757 (+ 523us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.644856 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 18:37:40.645788 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.633187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:34611 (local address 127.27.251.195:51085)
0808 18:37:40.633699 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.633758 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.635439 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.635877 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.635883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.636223 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:40.636882 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.636899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.637659 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.637662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.638680 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.638690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.639185 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.639188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.639342 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.639871 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:40.639888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:40.641464 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:40.643117 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.643123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.643167 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.643454 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.643729 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:40.643732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:40.643733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:40.643865 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:40.644255 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:40.644263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:40.644533 (+ 270us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.644685 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.644930 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260808 18:37:40.647082 15027 heartbeater.cc:347] Connected to a master server at 127.27.251.254:34611
I20260808 18:37:40.647351 15027 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:40.647894 15027 heartbeater.cc:511] Master 127.27.251.254:34611 requested a full tablet report, sending...
I20260808 18:37:40.648902 14532 ts_manager.cc:194] Registered new tserver with Master: d2d4d844ade5419c96f97d0eecbf582a (127.27.251.195:34267)
I20260808 18:37:40.649469 14532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:51085
I20260808 18:37:40.659153 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:40.668728 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.661308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.27.251.254:34611)
0808 18:37:40.661481 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.661487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.661644 (+ 157us) server_negotiation.cc:407] Connection header received
0808 18:37:40.661812 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.661816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.661867 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.661952 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:40.662682 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.663204 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.664065 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.664292 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.664405 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.665755 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:40.665759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:40.665762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:40.665789 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:40.666956 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.667406 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.667412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.667415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.667509 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:40.667801 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:40.667804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:40.667806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:40.668028 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:40.668202 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.668415 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.668563 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 18:37:40.671450 14532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:40.674571 14532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:40.689648 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.684300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.27.251.193:44505)
0808 18:37:40.684852 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.684866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.684885 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:37:40.684959 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.684963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.685105 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.685266 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:40.685683 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.686466 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.688092 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.688922 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689073 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689226 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.689390 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.689495 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":108,"threads_started":1}
I20260808 18:37:40.689810 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.683398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:34267 (local address 127.0.0.1:57556)
0808 18:37:40.683984 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.684036 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.684193 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.685352 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.685355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.685368 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:40.685570 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.685577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.687075 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.687079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.688198 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.688206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.689382 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.689493 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689516 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.689569 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.689652 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260808 18:37:40.689960 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.683556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:36797 (local address 127.0.0.1:49954)
0808 18:37:40.684445 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.684463 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.684604 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.685408 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.685412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.685426 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:40.685585 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.685592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.687110 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.687113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.688321 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.688333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.689578 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.689659 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689688 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.689756 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.689813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":41,"spinlock_wait_cycles":2048,"thread_start_us":135,"threads_started":1}
I20260808 18:37:40.689642 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.684115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:44505 (local address 127.0.0.1:53930)
0808 18:37:40.684456 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.684470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.684600 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.685257 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.685266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.685295 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:40.685568 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.685577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.686607 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.686613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.687952 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.687963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.689058 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.689163 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689199 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.689353 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.689492 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":98,"spinlock_wait_cycles":10880,"thread_start_us":94,"threads_started":1}
I20260808 18:37:40.690272 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.684236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.27.251.195:34267)
0808 18:37:40.684861 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.684873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.684896 (+ 23us) server_negotiation.cc:407] Connection header received
0808 18:37:40.684976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.684981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.685199 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.685473 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:40.685722 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.686920 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.688316 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.689235 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689768 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689981 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.690038 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.690139 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":161,"threads_started":1}
I20260808 18:37:40.690395 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.683799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.27.251.194:36797)
0808 18:37:40.684759 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.684772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.684796 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:37:40.684874 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.684881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.685091 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.685266 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:40.685730 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.686920 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.688460 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.689263 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.689383 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.689516 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.690155 (+ 639us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.690247 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":159,"threads_started":1}
I20260808 18:37:40.693773 14668 tablet_service.cc:1467] Processing CreateTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd (DEFAULT_TABLE table=test-table [id=473de8ef3c8b43b1a644291e3632733e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:40.693625 14810 tablet_service.cc:1467] Processing CreateTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd (DEFAULT_TABLE table=test-table [id=473de8ef3c8b43b1a644291e3632733e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:40.693625 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd (DEFAULT_TABLE table=test-table [id=473de8ef3c8b43b1a644291e3632733e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:40.694775 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f5632e8d004dbc91a03a44adefd1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:40.694775 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f5632e8d004dbc91a03a44adefd1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:40.694775 14668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f5632e8d004dbc91a03a44adefd1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:40.701560 15047 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: Bootstrap starting.
I20260808 18:37:40.703411 15048 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Bootstrap starting.
I20260808 18:37:40.704087 15047 tablet_bootstrap.cc:654] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:40.704819 15049 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Bootstrap starting.
I20260808 18:37:40.705227 15047 log.cc:824] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:40.705718 15048 tablet_bootstrap.cc:654] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:40.707015 15049 tablet_bootstrap.cc:654] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:40.707031 15048 log.cc:824] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:40.707847 15047 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: No bootstrap required, opened a new log
I20260808 18:37:40.708084 15047 ts_tablet_manager.cc:1397] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:37:40.708582 15049 log.cc:824] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:40.709306 15048 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: No bootstrap required, opened a new log
I20260808 18:37:40.709506 15048 ts_tablet_manager.cc:1397] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:37:40.710335 15049 tablet_bootstrap.cc:492] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: No bootstrap required, opened a new log
I20260808 18:37:40.710536 15049 ts_tablet_manager.cc:1397] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:37:40.713013 15047 raft_consensus.cc:348] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.713320 15047 raft_consensus.cc:374] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:40.713389 15047 raft_consensus.cc:729] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64762e84694b49a794059f70f159dc8e, State: Initialized, Role: FOLLOWER
I20260808 18:37:40.713444 15048 raft_consensus.cc:348] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.713690 15048 raft_consensus.cc:374] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:40.713747 15048 raft_consensus.cc:729] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4d844ade5419c96f97d0eecbf582a, State: Initialized, Role: FOLLOWER
I20260808 18:37:40.713953 15047 consensus_queue.cc:260] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.713794 15049 raft_consensus.cc:348] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.714123 15049 raft_consensus.cc:374] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:40.714181 15049 raft_consensus.cc:729] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c09d785b0804b9090dc005ac9540e3b, State: Initialized, Role: FOLLOWER
I20260808 18:37:40.714135 15048 consensus_queue.cc:260] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.714623 15049 consensus_queue.cc:260] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.715036 15047 ts_tablet_manager.cc:1428] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 18:37:40.715035 15048 ts_tablet_manager.cc:1428] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:40.715396 15027 heartbeater.cc:503] Master 127.27.251.254:34611 was elected leader, sending a full tablet report...
I20260808 18:37:40.715584 14881 heartbeater.cc:503] Master 127.27.251.254:34611 was elected leader, sending a full tablet report...
I20260808 18:37:40.715749 15049 ts_tablet_manager.cc:1428] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:40.716683 14738 heartbeater.cc:503] Master 127.27.251.254:34611 was elected leader, sending a full tablet report...
I20260808 18:37:40.799569 15054 raft_consensus.cc:482] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:40.799863 15054 raft_consensus.cc:504] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.801273 15054 leader_election.cc:288] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c09d785b0804b9090dc005ac9540e3b (127.27.251.194:36797), 64762e84694b49a794059f70f159dc8e (127.27.251.193:44505)
I20260808 18:37:40.805568 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.801709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:42889 (local address 127.27.251.193:44505)
0808 18:37:40.801844 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.801848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.802105 (+ 257us) server_negotiation.cc:407] Connection header received
0808 18:37:40.802205 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.802208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.802263 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.802337 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:40.802615 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.803272 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.804494 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.804944 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.805074 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.805143 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.805375 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.805448 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:37:40.805666 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.801518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:36797 (local address 127.27.251.195:42087)
0808 18:37:40.801744 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.801761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.801866 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.802236 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.802238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.802254 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:40.802439 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.802446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.803387 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.803393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.804482 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.804493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.805171 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.805175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.805333 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.805351 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.805439 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.805512 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":203}
I20260808 18:37:40.805666 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.801677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:42087 (local address 127.27.251.194:36797)
0808 18:37:40.801813 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.801818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.801834 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:37:40.802008 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.802026 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.802113 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.802214 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:40.802601 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.803266 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.804630 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.805040 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.805171 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.805234 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.805458 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.805537 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:37:40.805583 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.801574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:44505 (local address 127.27.251.195:42889)
0808 18:37:40.802005 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.802016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:40.802096 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:40.802365 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:40.802369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:40.802387 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:40.802508 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.802517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.803413 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.803419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.804367 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:40.804377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.805067 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.805071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:40.805172 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.805254 (+ 82us) client_negotiation.cc:769] Sending connection context
0808 18:37:40.805356 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 18:37:40.805429 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":115,"thread_start_us":155,"threads_started":1}
I20260808 18:37:40.806314 14830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f5632e8d004dbc91a03a44adefd1bd" candidate_uuid: "d2d4d844ade5419c96f97d0eecbf582a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c09d785b0804b9090dc005ac9540e3b" is_pre_election: true
I20260808 18:37:40.806664 14830 raft_consensus.cc:2454] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4d844ade5419c96f97d0eecbf582a in term 0.
I20260808 18:37:40.806700 14690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f5632e8d004dbc91a03a44adefd1bd" candidate_uuid: "d2d4d844ade5419c96f97d0eecbf582a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64762e84694b49a794059f70f159dc8e" is_pre_election: true
I20260808 18:37:40.806985 14690 raft_consensus.cc:2454] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4d844ade5419c96f97d0eecbf582a in term 0.
I20260808 18:37:40.807205 14906 leader_election.cc:302] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c09d785b0804b9090dc005ac9540e3b, d2d4d844ade5419c96f97d0eecbf582a; no voters:
I20260808 18:37:40.807559 15054 raft_consensus.cc:2781] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:40.807653 15054 raft_consensus.cc:482] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:40.807698 15054 raft_consensus.cc:3037] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:40.808790 15054 raft_consensus.cc:504] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.809134 15054 leader_election.cc:288] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [CANDIDATE]: Term 1 election: Requested vote from peers 3c09d785b0804b9090dc005ac9540e3b (127.27.251.194:36797), 64762e84694b49a794059f70f159dc8e (127.27.251.193:44505)
I20260808 18:37:40.809445 14830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f5632e8d004dbc91a03a44adefd1bd" candidate_uuid: "d2d4d844ade5419c96f97d0eecbf582a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c09d785b0804b9090dc005ac9540e3b"
I20260808 18:37:40.809536 14690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f5632e8d004dbc91a03a44adefd1bd" candidate_uuid: "d2d4d844ade5419c96f97d0eecbf582a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64762e84694b49a794059f70f159dc8e"
I20260808 18:37:40.809576 14830 raft_consensus.cc:3037] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:40.809669 14690 raft_consensus.cc:3037] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:40.810967 14690 raft_consensus.cc:2454] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4d844ade5419c96f97d0eecbf582a in term 1.
I20260808 18:37:40.810967 14830 raft_consensus.cc:2454] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4d844ade5419c96f97d0eecbf582a in term 1.
I20260808 18:37:40.811303 14907 leader_election.cc:302] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64762e84694b49a794059f70f159dc8e, d2d4d844ade5419c96f97d0eecbf582a; no voters:
I20260808 18:37:40.811534 15054 raft_consensus.cc:2781] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:40.811762 15054 raft_consensus.cc:686] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 1 LEADER]: Becoming Leader. State: Replica: d2d4d844ade5419c96f97d0eecbf582a, State: Running, Role: LEADER
I20260808 18:37:40.812153 15054 consensus_queue.cc:237] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } }
I20260808 18:37:40.815754 14530 catalog_manager.cc:5495] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d4d844ade5419c96f97d0eecbf582a (127.27.251.195). New cstate: current_term: 1 leader_uuid: "d2d4d844ade5419c96f97d0eecbf582a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d4d844ade5419c96f97d0eecbf582a" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 } health_report { overall_health: UNKNOWN } } }
I20260808 18:37:40.842549 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.838871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.27.251.195:34267)
0808 18:37:40.839030 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:40.839035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:40.839054 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:37:40.839126 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:40.839128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:40.839184 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:40.839298 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:40.839559 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.840078 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.841030 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:40.841250 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:40.841360 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:40.841679 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:40.842107 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:40.842246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:40.842301 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 18:37:40.842358 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:37:40.848738 14830 raft_consensus.cc:1261] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 1 FOLLOWER]: Refusing update from remote peer d2d4d844ade5419c96f97d0eecbf582a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:40.848744 14690 raft_consensus.cc:1261] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e [term 1 FOLLOWER]: Refusing update from remote peer d2d4d844ade5419c96f97d0eecbf582a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:40.849613 15057 consensus_queue.cc:1035] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64762e84694b49a794059f70f159dc8e" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:40.849835 15054 consensus_queue.cc:1035] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c09d785b0804b9090dc005ac9540e3b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:40.858757 15063 mvcc.cc:205] Tried to move back new op lower bound from 7316333612428722176 to 7316333612289646592. Current Snapshot: MvccSnapshot[applied={T|T < 7316333612428722176}]
I20260808 18:37:40.866921 14530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:40.867122 14530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:40.869827 14530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8eb30785c0d45ee99d0f0bb3a714747: Sending DeleteTablet for 3 replicas of tablet 61f5632e8d004dbc91a03a44adefd1bd
I20260808 18:37:40.870693 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:40 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:57556
I20260808 18:37:40.870730 14810 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:40 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:49954
I20260808 18:37:40.870728 14668 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f5632e8d004dbc91a03a44adefd1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:40 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:53930
I20260808 18:37:40.871292 15071 tablet_replica.cc:330] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: stopping tablet replica
I20260808 18:37:40.871454 15073 tablet_replica.cc:330] T 61f5632e8d004dbc91a03a44adefd1bd P 64762e84694b49a794059f70f159dc8e: stopping tablet replica
I20260808 18:37:40.871625 15071 raft_consensus.cc:2229] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:40.871904 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14603
I20260808 18:37:40.872033 15071 raft_consensus.cc:2258] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:40.873939 15071 ts_tablet_manager.cc:1905] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:40.876737 15072 tablet_replica.cc:330] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: stopping tablet replica
I20260808 18:37:40.877111 15072 raft_consensus.cc:2229] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:40.877458 15072 raft_consensus.cc:2258] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:40.877876 15071 ts_tablet_manager.cc:1918] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:40.877985 15071 log.cc:1196] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/61f5632e8d004dbc91a03a44adefd1bd
I20260808 18:37:40.878341 15071 ts_tablet_manager.cc:1939] T 61f5632e8d004dbc91a03a44adefd1bd P d2d4d844ade5419c96f97d0eecbf582a: Deleting consensus metadata
I20260808 18:37:40.879626 15072 ts_tablet_manager.cc:1905] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:40.879647 14516 catalog_manager.cc:4841] TS d2d4d844ade5419c96f97d0eecbf582a (127.27.251.195:34267): tablet 61f5632e8d004dbc91a03a44adefd1bd (table test-table [id=473de8ef3c8b43b1a644291e3632733e]) successfully deleted
I20260808 18:37:40.882229 15072 ts_tablet_manager.cc:1918] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:40.882352 15072 log.cc:1196] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/61f5632e8d004dbc91a03a44adefd1bd
I20260808 18:37:40.882649 15072 ts_tablet_manager.cc:1939] T 61f5632e8d004dbc91a03a44adefd1bd P 3c09d785b0804b9090dc005ac9540e3b: Deleting consensus metadata
I20260808 18:37:40.883870 14515 catalog_manager.cc:4841] TS 3c09d785b0804b9090dc005ac9540e3b (127.27.251.194:36797): tablet 61f5632e8d004dbc91a03a44adefd1bd (table test-table [id=473de8ef3c8b43b1a644291e3632733e]) successfully deleted
W20260808 18:37:40.887425 14516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:44505 (error 108)
I20260808 18:37:40.888276 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14743
I20260808 18:37:40.888593 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:40.888186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:44505 (local address 127.0.0.1:53934)
0808 18:37:40.888435 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:40.888498 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260808 18:37:40.889567 14516 catalog_manager.cc:4584] TS 64762e84694b49a794059f70f159dc8e (127.27.251.193:44505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61f5632e8d004dbc91a03a44adefd1bd: Network error: Client connection negotiation failed: client connection to 127.27.251.193:44505: connect: Connection refused (error 111)
I20260808 18:37:40.898130 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14887
I20260808 18:37:40.904866 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 14498
2026-08-08T18:37:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2446 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T18:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:40Z Disabled control of system clock
I20260808 18:37:40.939863 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45331
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:41.056088 15079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:41.056344 15079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:41.056380 15079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:41.060417 15079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:41.060483 15079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:41.060498 15079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:41.060541 15079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:41.060557 15079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:41.065325 15079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:41.066461 15079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:41.067855 15079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:41.076403 15088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:41.076385 15087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.076433 15079 server_base.cc:1034] running on GCE node
W20260808 18:37:41.076385 15090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.077373 15079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:41.078480 15079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:41.079685 15079 hybrid_clock.cc:648] HybridClock initialized: now 1786214261079676 us; error 34 us; skew 500 ppm
I20260808 18:37:41.081614 15079 webserver.cc:460] Webserver started at http://127.27.251.254:36767/ using document root <none> and password file <none>
I20260808 18:37:41.082224 15079 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:41.082319 15079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:41.082544 15079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:41.084686 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "804c5f0418f04b928eea004eb613eaec"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "5bc1cbeccd096ae917baf7208ef91291"
server_key_iv: "6624cf3ffdaee5fa8cabfecc77b2f4b0"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.085273 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "804c5f0418f04b928eea004eb613eaec"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "5bc1cbeccd096ae917baf7208ef91291"
server_key_iv: "6624cf3ffdaee5fa8cabfecc77b2f4b0"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.088980 15079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:37:41.091550 15097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.092896 15079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:41.093040 15079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "804c5f0418f04b928eea004eb613eaec"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "5bc1cbeccd096ae917baf7208ef91291"
server_key_iv: "6624cf3ffdaee5fa8cabfecc77b2f4b0"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.093161 15079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:41.107172 15079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:41.107831 15079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:41.108027 15079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:41.114976 15079 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45331
I20260808 18:37:41.114990 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45331 every 8 connection(s)
I20260808 18:37:41.115958 15079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:41.116428 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15079
I20260808 18:37:41.116597 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:41.116890 28655 external_mini_cluster.cc:1428] Setting key 71ebe1c6e72340c33d90dd0aa4d338bb
I20260808 18:37:41.119661 15158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:41.126142 15158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: Bootstrap starting.
I20260808 18:37:41.126555 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.118332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.27.251.254:45331)
0808 18:37:41.119031 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.119055 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.119104 (+ 49us) server_negotiation.cc:407] Connection header received
0808 18:37:41.119831 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.119848 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.120243 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.120575 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:41.120914 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.121876 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.123131 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.123621 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.123749 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.124686 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:41.124691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:41.124703 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:41.125103 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:41.125158 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:41.125176 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:41.125252 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.125467 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.125770 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":164,"threads_started":1}
I20260808 18:37:41.128655 15158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:41.129504 15158 log.cc:824] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:41.131394 15158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: No bootstrap required, opened a new log
I20260808 18:37:41.135214 15158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } }
I20260808 18:37:41.135424 15158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:41.135485 15158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804c5f0418f04b928eea004eb613eaec, State: Initialized, Role: FOLLOWER
I20260808 18:37:41.135893 15158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } }
I20260808 18:37:41.136008 15158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:41.136096 15158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:41.136214 15158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:41.137262 15158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } }
I20260808 18:37:41.137598 15158 leader_election.cc:302] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804c5f0418f04b928eea004eb613eaec; no voters:
I20260808 18:37:41.137908 15158 leader_election.cc:288] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:41.138037 15164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:41.138278 15164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [term 1 LEADER]: Becoming Leader. State: Replica: 804c5f0418f04b928eea004eb613eaec, State: Running, Role: LEADER
I20260808 18:37:41.138582 15164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } }
I20260808 18:37:41.139122 15158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:41.140240 15166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804c5f0418f04b928eea004eb613eaec. Latest consensus state: current_term: 1 leader_uuid: "804c5f0418f04b928eea004eb613eaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } } }
I20260808 18:37:41.140370 15166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:41.140578 15165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804c5f0418f04b928eea004eb613eaec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804c5f0418f04b928eea004eb613eaec" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45331 } } }
I20260808 18:37:41.140738 15165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:41.140793 15173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:41.143968 15173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:41.149235 15173 catalog_manager.cc:1347] Generated new cluster ID: db2c1ae1bbcf4a4eaffcc4ec49c68c5a
I20260808 18:37:41.149307 15173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:41.160112 15173 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:41.161258 15173 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:41.169147 15173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: Generated new TSK 0
I20260808 18:37:41.169890 15173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:41.182911 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--builtin_ntp_servers=127.27.251.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:41.314179 15184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:41.314440 15184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:41.314478 15184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:41.318394 15184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:41.318450 15184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:41.318504 15184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:41.323450 15184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:41.324577 15184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:41.325954 15184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:41.336002 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:41.336009 15189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.336066 15184 server_base.cc:1034] running on GCE node
W20260808 18:37:41.336009 15190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.336720 15184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:41.337388 15184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:41.338555 15184 hybrid_clock.cc:648] HybridClock initialized: now 1786214261338527 us; error 45 us; skew 500 ppm
I20260808 18:37:41.340477 15184 webserver.cc:460] Webserver started at http://127.27.251.193:37549/ using document root <none> and password file <none>
I20260808 18:37:41.341202 15184 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:41.341266 15184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:41.341521 15184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:41.343734 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "8cd93b4982c040be817e032af030745a"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "c0f22780f703dc041ca6f649b6ae1290"
server_key_iv: "a88064a14ed86b5b166c010fabf18b4d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.344309 15184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "8cd93b4982c040be817e032af030745a"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "c0f22780f703dc041ca6f649b6ae1290"
server_key_iv: "a88064a14ed86b5b166c010fabf18b4d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.348007 15184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:37:41.350481 15199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.351729 15184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:41.351866 15184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "8cd93b4982c040be817e032af030745a"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "c0f22780f703dc041ca6f649b6ae1290"
server_key_iv: "a88064a14ed86b5b166c010fabf18b4d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.351976 15184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:41.363507 15184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:41.364148 15184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:41.364344 15184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:41.365005 15184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:41.365936 15184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:41.366010 15184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.366081 15184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:41.366123 15184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.375497 15184 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:32967
I20260808 18:37:41.375536 15323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:32967 every 8 connection(s)
I20260808 18:37:41.376418 15184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:41.381714 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15184
I20260808 18:37:41.381855 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:41.382249 28655 external_mini_cluster.cc:1428] Setting key ead80daadd29f62e368cdc639c8438ba
I20260808 18:37:41.384393 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--builtin_ntp_servers=127.27.251.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:41.386696 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.378270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:56709 (local address 127.27.251.254:45331)
0808 18:37:41.378417 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.378420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.379462 (+ 1042us) server_negotiation.cc:407] Connection header received
0808 18:37:41.380950 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.380953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.381004 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.381079 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:41.382041 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.382863 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.384183 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.384595 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.384730 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.386035 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:41.386038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:41.386040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:41.386052 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:41.386072 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:41.386081 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:41.386141 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.386346 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.386456 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:37:41.387473 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.378630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45331 (local address 127.27.251.193:56709)
0808 18:37:41.379319 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.379360 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.380758 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.381257 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.381263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.381344 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:41.381855 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.381874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.382993 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.382997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.384034 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.384045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.384790 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.384797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.385029 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.385720 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:41.385724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:41.385782 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:41.385870 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:41.386169 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:41.386181 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:41.386190 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.386326 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.386633 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260808 18:37:41.388851 15324 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45331
I20260808 18:37:41.389106 15324 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:41.389793 15324 heartbeater.cc:511] Master 127.27.251.254:45331 requested a full tablet report, sending...
I20260808 18:37:41.391395 15113 ts_manager.cc:194] Registered new tserver with Master: 8cd93b4982c040be817e032af030745a (127.27.251.193:32967)
I20260808 18:37:41.392584 15113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.193:56709
W20260808 18:37:41.503230 15327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:41.503481 15327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:41.503545 15327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:41.507755 15327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:41.507804 15327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:41.507902 15327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:41.512648 15327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:41.513723 15327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:41.515132 15327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:41.524685 15335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.524951 15327 server_base.cc:1034] running on GCE node
W20260808 18:37:41.524694 15333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:41.524739 15332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.525454 15327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:41.526105 15327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:41.527292 15327 hybrid_clock.cc:648] HybridClock initialized: now 1786214261527279 us; error 25 us; skew 500 ppm
I20260808 18:37:41.529083 15327 webserver.cc:460] Webserver started at http://127.27.251.194:34699/ using document root <none> and password file <none>
I20260808 18:37:41.529683 15327 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:41.529773 15327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:41.529999 15327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:41.532075 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "421e3ed077bc479ea53a1b05927fb071"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "e976587d762a1692ffaa1b72500dae2c"
server_key_iv: "5d0e1082ea66cd8135b821e5b4aee96d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.532701 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "421e3ed077bc479ea53a1b05927fb071"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "e976587d762a1692ffaa1b72500dae2c"
server_key_iv: "5d0e1082ea66cd8135b821e5b4aee96d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.536262 15327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:41.538723 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.540082 15327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260808 18:37:41.540203 15327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "421e3ed077bc479ea53a1b05927fb071"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "e976587d762a1692ffaa1b72500dae2c"
server_key_iv: "5d0e1082ea66cd8135b821e5b4aee96d"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.540313 15327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:41.567121 15327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:41.567838 15327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:41.568039 15327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:41.568709 15327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:41.569684 15327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:41.569752 15327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.569823 15327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:41.569871 15327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.578541 15327 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:43635
I20260808 18:37:41.578572 15462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:43635 every 8 connection(s)
I20260808 18:37:41.579449 15327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:41.582376 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15327
I20260808 18:37:41.582504 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:41.582859 28655 external_mini_cluster.cc:1428] Setting key c35c72575c003cb8d58031587a278406
I20260808 18:37:41.585052 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--builtin_ntp_servers=127.27.251.212:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:41.590037 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.581339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:55239 (local address 127.27.251.254:45331)
0808 18:37:41.581479 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.581483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.582779 (+ 1296us) server_negotiation.cc:407] Connection header received
0808 18:37:41.584578 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.584581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.584629 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.584697 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:41.585931 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.586444 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.587775 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.588170 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.588295 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.589437 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:41.589444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:41.589450 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:41.589467 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:41.589496 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:41.589510 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:41.589617 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.589754 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.589833 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:37:41.590870 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.581731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45331 (local address 127.27.251.194:55239)
0808 18:37:41.582596 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.582656 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.584277 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.584918 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.584928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.585037 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:41.585774 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.585796 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.586580 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.586586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.587644 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.587654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.588322 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.588324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.588462 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.589171 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:41.589173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:41.589213 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:41.589277 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:41.589607 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:41.589614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:41.589619 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.589713 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.589929 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260808 18:37:41.592060 15463 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45331
I20260808 18:37:41.592329 15463 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:41.593029 15463 heartbeater.cc:511] Master 127.27.251.254:45331 requested a full tablet report, sending...
I20260808 18:37:41.594240 15113 ts_manager.cc:194] Registered new tserver with Master: 421e3ed077bc479ea53a1b05927fb071 (127.27.251.194:43635)
I20260808 18:37:41.595125 15113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.194:55239
W20260808 18:37:41.701408 15466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:41.701673 15466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:41.701717 15466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:41.705785 15466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:41.705858 15466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:41.705941 15466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:41.710790 15466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:41.711895 15466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:41.713398 15466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:41.723086 15475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.723295 15466 server_base.cc:1034] running on GCE node
W20260808 18:37:41.723101 15472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:41.723104 15473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:41.723896 15466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:41.724560 15466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:41.725726 15466 hybrid_clock.cc:648] HybridClock initialized: now 1786214261725713 us; error 25 us; skew 500 ppm
I20260808 18:37:41.727543 15466 webserver.cc:460] Webserver started at http://127.27.251.195:37831/ using document root <none> and password file <none>
I20260808 18:37:41.728118 15466 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:41.728180 15466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:41.728443 15466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:41.730509 15466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "be1d1afda0634305bff4c0960f17e2d5"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "4226ab3e139201c194fd3d40d994381a"
server_key_iv: "f2950a9268603d690fb1619c4a5c5c81"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.731081 15466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "be1d1afda0634305bff4c0960f17e2d5"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "4226ab3e139201c194fd3d40d994381a"
server_key_iv: "f2950a9268603d690fb1619c4a5c5c81"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.734838 15466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260808 18:37:41.737242 15481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.738435 15466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:37:41.738555 15466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "be1d1afda0634305bff4c0960f17e2d5"
format_stamp: "Formatted at 2026-08-08 18:37:41 on dist-test-slave-fd7l"
server_key: "4226ab3e139201c194fd3d40d994381a"
server_key_iv: "f2950a9268603d690fb1619c4a5c5c81"
server_key_version: "encryptionkey@0"
I20260808 18:37:41.738665 15466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:41.751117 15466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:41.751737 15466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:41.751924 15466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:41.752583 15466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:41.753505 15466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:41.753551 15466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.753589 15466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:41.753656 15466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:41.762771 15466 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:44491
I20260808 18:37:41.762804 15601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:44491 every 8 connection(s)
I20260808 18:37:41.763688 15466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:41.772977 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15466
I20260808 18:37:41.773087 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.765662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:50717 (local address 127.27.251.254:45331)
0808 18:37:41.765812 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.765816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.766853 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 18:37:41.768323 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.768327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.768388 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.768498 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:41.769425 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.769968 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.771122 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.771355 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.771472 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.772546 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:41.772553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:41.772559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:41.772573 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:41.772603 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:41.772612 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:41.772704 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.772874 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.772941 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 18:37:41.773108 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:41.773406 28655 external_mini_cluster.cc:1428] Setting key 680c811439b82bebbed7176af3be1230
I20260808 18:37:41.774052 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.766046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45331 (local address 127.27.251.195:50717)
0808 18:37:41.766699 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.766741 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.768074 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.768702 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.768710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.768793 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:41.769275 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.769287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.770104 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.770108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.770985 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.770994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.771518 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.771522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.771688 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.772260 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:41.772261 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:41.772300 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:41.772361 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:41.772726 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:41.772742 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:41.772746 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.772873 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.773158 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260808 18:37:41.775342 15602 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45331
I20260808 18:37:41.775579 15602 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:41.776121 15602 heartbeater.cc:511] Master 127.27.251.254:45331 requested a full tablet report, sending...
I20260808 18:37:41.777119 15113 ts_manager.cc:194] Registered new tserver with Master: be1d1afda0634305bff4c0960f17e2d5 (127.27.251.195:44491)
I20260808 18:37:41.777719 15113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.195:50717
I20260808 18:37:41.787662 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:41.793659 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.789875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.27.251.254:45331)
0808 18:37:41.790055 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.790059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.790168 (+ 109us) server_negotiation.cc:407] Connection header received
0808 18:37:41.790270 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.790272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.790319 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.790386 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:41.790681 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.791176 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.792049 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.792269 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.792345 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.793267 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:41.793270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:41.793272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:41.793283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:41.793298 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:41.793306 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:41.793362 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.793464 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.793510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 18:37:41.796533 15113 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:41.799662 15113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:41.815202 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.808860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.27.251.194:43635)
0808 18:37:41.809340 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.809361 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.810098 (+ 737us) server_negotiation.cc:407] Connection header received
0808 18:37:41.810205 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.810210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.810460 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.810629 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:41.811037 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.812415 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.813840 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.814412 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.814567 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.814771 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.814901 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.815018 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":140,"threads_started":1}
I20260808 18:37:41.815194 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.808733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43635 (local address 127.0.0.1:44604)
0808 18:37:41.809323 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.809377 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.810022 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.810639 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.810649 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.810673 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:41.810852 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.810860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.812616 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.812621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.813689 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.813701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.814566 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.814571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.814692 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.814735 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.814864 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.815026 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":81,"threads_started":1}
I20260808 18:37:41.816608 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.808883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:44491 (local address 127.0.0.1:52874)
0808 18:37:41.809496 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.809512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.809634 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.810839 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.810842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.810860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:41.811139 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.811147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.812677 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.812680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.815094 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.815106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.816200 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.816205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.816300 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.816328 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.816378 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.816451 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":1196,"thread_start_us":120,"threads_started":1}
I20260808 18:37:41.817008 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.809508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:32967 (local address 127.0.0.1:51790)
0808 18:37:41.809881 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.809894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.809973 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.810807 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.810810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.810850 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:41.811139 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.811147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.813652 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.813658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.815094 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.815106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.816558 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.816563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.816663 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.816690 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.816783 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.816855 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":193,"spinlock_wait_cycles":3328,"thread_start_us":103,"threads_started":1}
I20260808 18:37:41.817327 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.809957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.27.251.195:44491)
0808 18:37:41.810328 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.810338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.810359 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:37:41.810426 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.810449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.810667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.811002 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:41.811281 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.812548 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.815246 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.816050 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.816865 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.817000 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.817041 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.817148 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":152,"threads_started":1}
I20260808 18:37:41.817473 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.809695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.27.251.193:32967)
0808 18:37:41.810284 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.810295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.810315 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:37:41.810381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.810386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.810566 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.810711 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:41.812608 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.813534 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.815246 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.816312 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.816437 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.816627 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.817200 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.817306 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":100,"threads_started":1}
I20260808 18:37:41.819085 15390 tablet_service.cc:1467] Processing CreateTablet for tablet 436383f989de409ba96a4a510fdf489a (DEFAULT_TABLE table=test-table [id=12968167c29f44a6827df9be6f699eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:41.819816 15531 tablet_service.cc:1467] Processing CreateTablet for tablet 436383f989de409ba96a4a510fdf489a (DEFAULT_TABLE table=test-table [id=12968167c29f44a6827df9be6f699eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:41.820328 15390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436383f989de409ba96a4a510fdf489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:41.820277 15251 tablet_service.cc:1467] Processing CreateTablet for tablet 436383f989de409ba96a4a510fdf489a (DEFAULT_TABLE table=test-table [id=12968167c29f44a6827df9be6f699eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:41.820874 15531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436383f989de409ba96a4a510fdf489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:41.821367 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436383f989de409ba96a4a510fdf489a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:41.826799 15622 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Bootstrap starting.
I20260808 18:37:41.828225 15623 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Bootstrap starting.
I20260808 18:37:41.828907 15624 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Bootstrap starting.
I20260808 18:37:41.829406 15622 tablet_bootstrap.cc:654] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:41.830418 15622 log.cc:824] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:41.830482 15623 tablet_bootstrap.cc:654] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:41.831197 15624 tablet_bootstrap.cc:654] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:41.831214 15623 log.cc:824] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:41.832067 15624 log.cc:824] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:41.832643 15622 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: No bootstrap required, opened a new log
I20260808 18:37:41.832844 15622 ts_tablet_manager.cc:1397] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:37:41.833364 15623 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: No bootstrap required, opened a new log
I20260808 18:37:41.833586 15623 ts_tablet_manager.cc:1397] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:37:41.834062 15624 tablet_bootstrap.cc:492] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: No bootstrap required, opened a new log
I20260808 18:37:41.834264 15624 ts_tablet_manager.cc:1397] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:37:41.837265 15622 raft_consensus.cc:348] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.837545 15622 raft_consensus.cc:374] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:41.837603 15622 raft_consensus.cc:729] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d1afda0634305bff4c0960f17e2d5, State: Initialized, Role: FOLLOWER
I20260808 18:37:41.837774 15623 raft_consensus.cc:348] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.838018 15623 raft_consensus.cc:374] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:41.838073 15623 raft_consensus.cc:729] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421e3ed077bc479ea53a1b05927fb071, State: Initialized, Role: FOLLOWER
I20260808 18:37:41.838110 15622 consensus_queue.cc:260] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.838068 15624 raft_consensus.cc:348] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.838361 15624 raft_consensus.cc:374] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:41.838426 15624 raft_consensus.cc:729] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd93b4982c040be817e032af030745a, State: Initialized, Role: FOLLOWER
I20260808 18:37:41.838537 15623 consensus_queue.cc:260] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.838999 15602 heartbeater.cc:503] Master 127.27.251.254:45331 was elected leader, sending a full tablet report...
I20260808 18:37:41.838943 15624 consensus_queue.cc:260] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.839138 15622 ts_tablet_manager.cc:1428] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:37:41.839511 15623 ts_tablet_manager.cc:1428] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 18:37:41.839659 15463 heartbeater.cc:503] Master 127.27.251.254:45331 was elected leader, sending a full tablet report...
I20260808 18:37:41.840044 15624 ts_tablet_manager.cc:1428] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 18:37:41.840368 15324 heartbeater.cc:503] Master 127.27.251.254:45331 was elected leader, sending a full tablet report...
W20260808 18:37:41.877640 15325 tablet.cc:2369] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:41.915712 15628 raft_consensus.cc:482] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:41.915947 15628 raft_consensus.cc:504] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.917291 15628 leader_election.cc:288] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 421e3ed077bc479ea53a1b05927fb071 (127.27.251.194:43635), 8cd93b4982c040be817e032af030745a (127.27.251.193:32967)
I20260808 18:37:41.921463 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.917718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:46833 (local address 127.27.251.193:32967)
0808 18:37:41.917857 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.917864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.918062 (+ 198us) server_negotiation.cc:407] Connection header received
0808 18:37:41.918188 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.918191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.918239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.918322 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:41.918638 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.919127 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920414 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.920836 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920970 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.921036 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.921234 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.921322 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:37:41.921466 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.917488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43635 (local address 127.27.251.195:43225)
0808 18:37:41.917655 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.917671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.917774 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.918173 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.918176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.918197 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:41.918339 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.918347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.919262 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.919265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.920268 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.920278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.920994 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.921152 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.921198 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.921277 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.921340 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":98}
I20260808 18:37:41.921504 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.917581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:32967 (local address 127.27.251.195:46833)
0808 18:37:41.917964 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:41.917977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:41.918081 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:41.918354 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:41.918357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:41.918375 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:41.918518 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.918526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.919249 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.919255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.920268 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:41.920278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.920968 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:41.921078 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.921113 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 18:37:41.921197 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:37:41.921267 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":137,"thread_start_us":132,"threads_started":1}
I20260808 18:37:41.921895 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.917746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:43225 (local address 127.27.251.194:43635)
0808 18:37:41.917889 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.917894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.917911 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:37:41.917972 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.917976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.918044 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.918128 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:41.918497 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.919128 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920397 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.920861 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.920994 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.921063 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.921312 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.921486 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 18:37:41.922303 15274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436383f989de409ba96a4a510fdf489a" candidate_uuid: "be1d1afda0634305bff4c0960f17e2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd93b4982c040be817e032af030745a" is_pre_election: true
I20260808 18:37:41.922582 15411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436383f989de409ba96a4a510fdf489a" candidate_uuid: "be1d1afda0634305bff4c0960f17e2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421e3ed077bc479ea53a1b05927fb071" is_pre_election: true
I20260808 18:37:41.922713 15274 raft_consensus.cc:2454] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1d1afda0634305bff4c0960f17e2d5 in term 0.
I20260808 18:37:41.922899 15411 raft_consensus.cc:2454] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1d1afda0634305bff4c0960f17e2d5 in term 0.
I20260808 18:37:41.923333 15483 leader_election.cc:302] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cd93b4982c040be817e032af030745a, be1d1afda0634305bff4c0960f17e2d5; no voters:
I20260808 18:37:41.923591 15628 raft_consensus.cc:2781] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:41.923678 15628 raft_consensus.cc:482] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:41.923722 15628 raft_consensus.cc:3037] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:41.924974 15628 raft_consensus.cc:504] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.925298 15628 leader_election.cc:288] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [CANDIDATE]: Term 1 election: Requested vote from peers 421e3ed077bc479ea53a1b05927fb071 (127.27.251.194:43635), 8cd93b4982c040be817e032af030745a (127.27.251.193:32967)
I20260808 18:37:41.925638 15411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436383f989de409ba96a4a510fdf489a" candidate_uuid: "be1d1afda0634305bff4c0960f17e2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "421e3ed077bc479ea53a1b05927fb071"
I20260808 18:37:41.925645 15274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436383f989de409ba96a4a510fdf489a" candidate_uuid: "be1d1afda0634305bff4c0960f17e2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd93b4982c040be817e032af030745a"
I20260808 18:37:41.925773 15411 raft_consensus.cc:3037] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:41.925809 15274 raft_consensus.cc:3037] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:41.926935 15274 raft_consensus.cc:2454] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1d1afda0634305bff4c0960f17e2d5 in term 1.
I20260808 18:37:41.927017 15411 raft_consensus.cc:2454] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1d1afda0634305bff4c0960f17e2d5 in term 1.
I20260808 18:37:41.927250 15483 leader_election.cc:302] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cd93b4982c040be817e032af030745a, be1d1afda0634305bff4c0960f17e2d5; no voters:
I20260808 18:37:41.927480 15628 raft_consensus.cc:2781] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:41.927748 15628 raft_consensus.cc:686] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 1 LEADER]: Becoming Leader. State: Replica: be1d1afda0634305bff4c0960f17e2d5, State: Running, Role: LEADER
I20260808 18:37:41.928117 15628 consensus_queue.cc:237] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } }
I20260808 18:37:41.931635 15115 catalog_manager.cc:5495] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to be1d1afda0634305bff4c0960f17e2d5 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "be1d1afda0634305bff4c0960f17e2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be1d1afda0634305bff4c0960f17e2d5" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 44491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 } health_report { overall_health: UNKNOWN } } }
I20260808 18:37:41.965904 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:41.962092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.27.251.195:44491)
0808 18:37:41.962252 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:41.962257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:41.962285 (+ 28us) server_negotiation.cc:407] Connection header received
0808 18:37:41.962385 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:41.962387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:41.962443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:41.962568 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:41.962842 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.963372 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.964265 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:41.964493 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:41.964634 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:41.964973 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:41.965460 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:41.965602 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:41.965664 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 18:37:41.965723 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 18:37:41.972043 15411 raft_consensus.cc:1261] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 1 FOLLOWER]: Refusing update from remote peer be1d1afda0634305bff4c0960f17e2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:41.972043 15274 raft_consensus.cc:1261] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 1 FOLLOWER]: Refusing update from remote peer be1d1afda0634305bff4c0960f17e2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:41.972838 15633 consensus_queue.cc:1035] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "421e3ed077bc479ea53a1b05927fb071" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:41.973026 15628 consensus_queue.cc:1035] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd93b4982c040be817e032af030745a" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:41.980677 15636 mvcc.cc:205] Tried to move back new op lower bound from 7316333617029586944 to 7316333616860475392. Current Snapshot: MvccSnapshot[applied={T|T < 7316333617029586944}]
I20260808 18:37:41.983896 15638 mvcc.cc:205] Tried to move back new op lower bound from 7316333617029586944 to 7316333616860475392. Current Snapshot: MvccSnapshot[applied={T|T < 7316333617029586944}]
I20260808 18:37:41.989765 15113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:41.990134 15113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:41.993216 15113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 804c5f0418f04b928eea004eb613eaec: Sending DeleteTablet for 3 replicas of tablet 436383f989de409ba96a4a510fdf489a
I20260808 18:37:41.994462 15531 tablet_service.cc:1514] Processing DeleteTablet for tablet 436383f989de409ba96a4a510fdf489a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:41 UTC) from {username='slave'} at 127.0.0.1:52874
I20260808 18:37:41.994522 15390 tablet_service.cc:1514] Processing DeleteTablet for tablet 436383f989de409ba96a4a510fdf489a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:41 UTC) from {username='slave'} at 127.0.0.1:44604
I20260808 18:37:41.995383 15646 tablet_replica.cc:330] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: stopping tablet replica
I20260808 18:37:41.995525 15251 tablet_service.cc:1514] Processing DeleteTablet for tablet 436383f989de409ba96a4a510fdf489a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:41 UTC) from {username='slave'} at 127.0.0.1:51790
I20260808 18:37:41.995862 15646 raft_consensus.cc:2229] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:41.996011 15648 tablet_replica.cc:330] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: stopping tablet replica
I20260808 18:37:41.996336 15648 raft_consensus.cc:2229] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:41.996367 15646 raft_consensus.cc:2258] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:41.996707 15648 raft_consensus.cc:2258] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:41.998268 15648 ts_tablet_manager.cc:1905] T 436383f989de409ba96a4a510fdf489a P 8cd93b4982c040be817e032af030745a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:41.998840 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15184
I20260808 18:37:42.001559 15646 ts_tablet_manager.cc:1905] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:42.005553 15646 ts_tablet_manager.cc:1918] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:42.005939 15647 tablet_replica.cc:330] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: stopping tablet replica
I20260808 18:37:42.006410 15647 raft_consensus.cc:2229] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:42.006876 15647 raft_consensus.cc:2258] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:42.008771 15647 ts_tablet_manager.cc:1905] T 436383f989de409ba96a4a510fdf489a P 421e3ed077bc479ea53a1b05927fb071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:42.008988 15646 log.cc:1196] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/436383f989de409ba96a4a510fdf489a
I20260808 18:37:42.009444 15646 ts_tablet_manager.cc:1939] T 436383f989de409ba96a4a510fdf489a P be1d1afda0634305bff4c0960f17e2d5: Deleting consensus metadata
W20260808 18:37:42.010811 15099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:32967 (error 108)
I20260808 18:37:42.011065 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15327
I20260808 18:37:42.011338 15099 catalog_manager.cc:4841] TS be1d1afda0634305bff4c0960f17e2d5 (127.27.251.195:44491): tablet 436383f989de409ba96a4a510fdf489a (table test-table [id=12968167c29f44a6827df9be6f699eb7]) successfully deleted
I20260808 18:37:42.012885 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:42.012456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:32967 (local address 127.0.0.1:51802)
0808 18:37:42.012701 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:42.012791 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260808 18:37:42.013106 15099 catalog_manager.cc:4584] TS 8cd93b4982c040be817e032af030745a (127.27.251.193:32967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 436383f989de409ba96a4a510fdf489a: Network error: Client connection negotiation failed: client connection to 127.27.251.193:32967: connect: Connection refused (error 111)
I20260808 18:37:42.021739 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:42.021279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43635 (local address 127.0.0.1:44608)
0808 18:37:42.021540 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:42.021618 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.194:43635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260808 18:37:42.021895 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15466
I20260808 18:37:42.032714 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15079
2026-08-08T18:37:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1130 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:42 dist-test-slave-fd7l krb5kdc[15652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:42 dist-test-slave-fd7l krb5kdc[15652](info): set up 2 sockets
Aug 08 18:37:42 dist-test-slave-fd7l krb5kdc[15652](info): commencing operation
krb5kdc: starting...
W20260808 18:37:43.179055 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:43 dist-test-slave-fd7l krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:37:43.315225 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45285
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:45285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:43.430599 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:43.430896 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:43.430979 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:43.434952 15666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:43.435034 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:43.435070 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:43.435110 15666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:43.435145 15666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:43.440253 15666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:45285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:45285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:43.441288 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:43.442322 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:43.450981 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:43.450994 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:43.451047 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:43.451241 15666 server_base.cc:1034] running on GCE node
I20260808 18:37:43.452024 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:43.453213 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:43.454406 15666 hybrid_clock.cc:648] HybridClock initialized: now 1786214263454388 us; error 27 us; skew 500 ppm
Aug 08 18:37:43 dist-test-slave-fd7l krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214263, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:43.457327 15666 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:37:43.458283 15666 webserver.cc:460] Webserver started at http://127.27.251.254:42545/ using document root <none> and password file <none>
I20260808 18:37:43.458863 15666 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:43.458954 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:43.459197 15666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:43.461356 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "80c4c3cb48e744f495148ea338c7a48e"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "631d4407f9827d2d7dd777dca758f35b"
server_key_iv: "595f0caefd0d162a12c9fcf050d24a12"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.461931 15666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "80c4c3cb48e744f495148ea338c7a48e"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "631d4407f9827d2d7dd777dca758f35b"
server_key_iv: "595f0caefd0d162a12c9fcf050d24a12"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.467267 15666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 18:37:43.470887 15681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:43.472436 15666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:37:43.472590 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "80c4c3cb48e744f495148ea338c7a48e"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "631d4407f9827d2d7dd777dca758f35b"
server_key_iv: "595f0caefd0d162a12c9fcf050d24a12"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.472707 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:43.485136 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:43.488091 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:43.488255 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:43.496078 15666 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:45285
I20260808 18:37:43.496106 15741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:45285 every 8 connection(s)
I20260808 18:37:43.497080 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:43.500105 15742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:43.502669 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15666
I20260808 18:37:43.502843 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:43.503228 28655 external_mini_cluster.cc:1428] Setting key 49376e2dd3a8570757fd5df68d72d971
I20260808 18:37:43.507108 15742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: Bootstrap starting.
I20260808 18:37:43.509685 15742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:43.510489 15742 log.cc:824] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:37:43 dist-test-slave-fd7l krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:43.512615 15742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: No bootstrap required, opened a new log
I20260808 18:37:43.516738 15742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } }
I20260808 18:37:43.517053 15742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:43.517148 15742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4c3cb48e744f495148ea338c7a48e, State: Initialized, Role: FOLLOWER
I20260808 18:37:43.517648 15742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } }
I20260808 18:37:43.517817 15742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:43.517923 15742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:43.518070 15742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:43.519311 15742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } }
I20260808 18:37:43.519707 15742 leader_election.cc:302] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80c4c3cb48e744f495148ea338c7a48e; no voters:
I20260808 18:37:43.520117 15742 leader_election.cc:288] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:43.520228 15747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:43.520500 15747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [term 1 LEADER]: Becoming Leader. State: Replica: 80c4c3cb48e744f495148ea338c7a48e, State: Running, Role: LEADER
I20260808 18:37:43.520833 15747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } }
I20260808 18:37:43.521672 15742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:43.522354 15748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80c4c3cb48e744f495148ea338c7a48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } } }
I20260808 18:37:43.522481 15748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:43.522921 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80c4c3cb48e744f495148ea338c7a48e. Latest consensus state: current_term: 1 leader_uuid: "80c4c3cb48e744f495148ea338c7a48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80c4c3cb48e744f495148ea338c7a48e" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 45285 } } }
I20260808 18:37:43.523064 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:43.524004 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:43.504820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.27.251.254:45285)
0808 18:37:43.505318 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:43.505334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:43.505370 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:37:43.506158 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:43.506178 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:43.506514 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:43.506861 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:43.507687 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.508741 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.509608 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.510154 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.510336 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:43.512932 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:43.512937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:43.512950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:43.512977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:43.519812 (+ 6835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:43.520564 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:43.520571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:43.520578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:43.520735 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:43.521119 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:43.521123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:43.521125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:43.521702 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:43.521901 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:43.522469 (+ 568us) server_negotiation.cc:299] Negotiation successful
0808 18:37:43.522863 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":149,"threads_started":1}
I20260808 18:37:43.524751 15753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:43.527557 15753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:43.532986 15753 catalog_manager.cc:1347] Generated new cluster ID: 51c36febee92486baf83c62e30986973
I20260808 18:37:43.533074 15753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:43.545354 15753 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:43.546423 15753 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:43.564693 15753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: Generated new TSK 0
I20260808 18:37:43.565511 15753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:37:43.608275 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:43.730690 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:43.730974 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:43.731021 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:43.734992 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:43.735106 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:43.735231 15768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:43.743988 15768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:43.745477 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:43.746796 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:43.757102 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:43.757108 15778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:43.757098 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:43.757330 15768 server_base.cc:1034] running on GCE node
I20260808 18:37:43.757858 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:43.758626 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:43.759840 15768 hybrid_clock.cc:648] HybridClock initialized: now 1786214263759809 us; error 44 us; skew 500 ppm
Aug 08 18:37:43 dist-test-slave-fd7l krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214263, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:43.762849 15768 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:37:43.763829 15768 webserver.cc:460] Webserver started at http://127.27.251.193:36699/ using document root <none> and password file <none>
I20260808 18:37:43.764379 15768 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:43.764439 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:43.764699 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:43.766868 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "a68b3dc1c5074865af71898d3918fc74"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "0c6e0245b0dc3b0ee4ea77e86a7271e5"
server_key_iv: "16c4389f6a6f8189f514db6bc3377a69"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.767423 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "a68b3dc1c5074865af71898d3918fc74"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "0c6e0245b0dc3b0ee4ea77e86a7271e5"
server_key_iv: "16c4389f6a6f8189f514db6bc3377a69"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.771487 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:37:43.774199 15785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:43.775534 15768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 18:37:43.775651 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "a68b3dc1c5074865af71898d3918fc74"
format_stamp: "Formatted at 2026-08-08 18:37:43 on dist-test-slave-fd7l"
server_key: "0c6e0245b0dc3b0ee4ea77e86a7271e5"
server_key_iv: "16c4389f6a6f8189f514db6bc3377a69"
server_key_version: "encryptionkey@0"
I20260808 18:37:43.775748 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:43.794862 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:43.798508 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:43.798728 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:43.799364 15768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:43.800359 15768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:43.800415 15768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:43.800475 15768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:43.800495 15768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:43.810802 15768 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:42723
I20260808 18:37:43.810918 15908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:42723 every 8 connection(s)
I20260808 18:37:43.811836 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:43.816880 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15768
I20260808 18:37:43.817034 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:43.817378 28655 external_mini_cluster.cc:1428] Setting key 2644286f9af61124cec05dc240585bcf
Aug 08 18:37:43 dist-test-slave-fd7l krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214263, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:43.828835 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:43.813913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:48599 (local address 127.27.251.254:45285)
0808 18:37:43.814091 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:43.814095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:43.814917 (+ 822us) server_negotiation.cc:407] Connection header received
0808 18:37:43.816290 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:43.816294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:43.816350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:43.816441 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:43.818161 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.818943 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.820789 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.821073 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.821186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:43.824389 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:43.824392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:43.824394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:43.824423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:43.826210 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:43.826962 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:43.826965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:43.826966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:43.827045 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:43.827539 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:43.827541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:43.827542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:43.827746 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:43.827826 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:43.828588 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 18:37:43.828705 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 18:37:43.830209 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:43.814293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45285 (local address 127.27.251.193:48599)
0808 18:37:43.814742 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:43.814787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:43.816040 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:43.816718 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:43.816729 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:43.817284 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:43.817985 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:43.818007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.819086 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.819089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:43.820608 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:43.820625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:43.821261 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:43.821269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:43.821526 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:43.822320 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:43.822337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:43.824181 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:43.826357 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:43.826365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:43.826386 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:43.826850 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:43.827152 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:43.827158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:43.827163 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:43.827399 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:43.827865 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:43.827873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:43.828254 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 18:37:43.828501 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 18:37:43.828953 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:37:43.831666 15909 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45285
I20260808 18:37:43.831938 15909 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:43.832548 15909 heartbeater.cc:511] Master 127.27.251.254:45285 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:37:43.834216 15697 ts_manager.cc:194] Registered new tserver with Master: a68b3dc1c5074865af71898d3918fc74 (127.27.251.193:42723)
I20260808 18:37:43.835462 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:48599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:37:43.852116 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:43.980623 15914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:43.980918 15914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:43.981026 15914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:43.985756 15914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:43.985832 15914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:43.985953 15914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:43.992210 15914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:43.993464 15914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:43.994603 15914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:44.005403 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.005370 15919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.005368 15920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:44.005447 15914 server_base.cc:1034] running on GCE node
I20260808 18:37:44.006139 15914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:44.006842 15914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:44.008095 15914 hybrid_clock.cc:648] HybridClock initialized: now 1786214264008058 us; error 51 us; skew 500 ppm
Aug 08 18:37:44 dist-test-slave-fd7l krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:44.011420 15914 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:37:44.012315 15914 webserver.cc:460] Webserver started at http://127.27.251.194:42855/ using document root <none> and password file <none>
I20260808 18:37:44.012964 15914 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:44.013058 15914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:44.013303 15914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:44.015347 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "abaf364c37d8464080035f1b2ea5a3ee"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "2db663323947582993a8a907fa513e9c"
server_key_iv: "e5a90a5a2e07d15b1118c956115415cb"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.015933 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "abaf364c37d8464080035f1b2ea5a3ee"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "2db663323947582993a8a907fa513e9c"
server_key_iv: "e5a90a5a2e07d15b1118c956115415cb"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.019852 15914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:37:44.022570 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.024034 15914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:37:44.024163 15914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "abaf364c37d8464080035f1b2ea5a3ee"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "2db663323947582993a8a907fa513e9c"
server_key_iv: "e5a90a5a2e07d15b1118c956115415cb"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.024268 15914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:44.035039 15914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:44.038004 15914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:44.038213 15914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:44.038793 15914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:44.039628 15914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:44.039703 15914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.039772 15914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:44.039826 15914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.049245 15914 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:39989
I20260808 18:37:44.049278 16054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:39989 every 8 connection(s)
I20260808 18:37:44.050292 15914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:44.059981 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 15914
I20260808 18:37:44.060141 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:44.060485 28655 external_mini_cluster.cc:1428] Setting key 079c4918136d7203b982832dd07b14b6
Aug 08 18:37:44 dist-test-slave-fd7l krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:44.065886 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.052247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:60721 (local address 127.27.251.254:45285)
0808 18:37:44.052406 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.052410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.053282 (+ 872us) server_negotiation.cc:407] Connection header received
0808 18:37:44.054832 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.054836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.054909 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.055027 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:44.056309 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.056960 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.058233 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.058474 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.058598 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.061552 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:44.061555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:44.061557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:44.061584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:44.063242 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.063938 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.063944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.063949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.064049 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.064534 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.064538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.064543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.064814 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:44.064943 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.065489 (+ 546us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.065652 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:37:44.066699 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.052650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45285 (local address 127.27.251.194:60721)
0808 18:37:44.053119 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.053165 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.054595 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.055226 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.055233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.055625 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:44.056144 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.056159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.057108 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.057113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.058090 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.058100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.058652 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.058658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.058840 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.059511 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:44.059529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:44.061356 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:44.063371 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:44.063381 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:44.063400 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:44.063816 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:44.064152 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:44.064159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:44.064165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:44.064391 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:44.064961 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:44.064967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:44.065233 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.065406 (+ 173us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.065701 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260808 18:37:44.068107 16055 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45285
I20260808 18:37:44.068384 16055 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:44.068976 16055 heartbeater.cc:511] Master 127.27.251.254:45285 requested a full tablet report, sending...
I20260808 18:37:44.070182 15697 ts_manager.cc:194] Registered new tserver with Master: abaf364c37d8464080035f1b2ea5a3ee (127.27.251.194:39989)
I20260808 18:37:44.071014 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:60721
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:37:44.091461 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:44.211123 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:44.211370 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:44.211413 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:44.215355 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:44.215411 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:44.215474 16061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:44.220889 16061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:45285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:44.221958 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:44.223126 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:44.233057 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.233147 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.233258 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:44.233601 16061 server_base.cc:1034] running on GCE node
I20260808 18:37:44.234021 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:44.234717 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:44.235926 16061 hybrid_clock.cc:648] HybridClock initialized: now 1786214264235906 us; error 29 us; skew 500 ppm
Aug 08 18:37:44 dist-test-slave-fd7l krb5kdc[15652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:44.238857 16061 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:37:44.239867 16061 webserver.cc:460] Webserver started at http://127.27.251.195:40143/ using document root <none> and password file <none>
I20260808 18:37:44.240455 16061 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:44.240574 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:44.240825 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:44.243037 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "e5448eb1446c4af9815836436609e870"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "1c92a07f0ba1b0e5b4272b4a113b20ba"
server_key_iv: "2a3ba27227c9e10dd86dba70fdab97e7"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.243626 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "e5448eb1446c4af9815836436609e870"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "1c92a07f0ba1b0e5b4272b4a113b20ba"
server_key_iv: "2a3ba27227c9e10dd86dba70fdab97e7"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.247604 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:37:44.250229 16077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.251585 16061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:44.251788 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "e5448eb1446c4af9815836436609e870"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "1c92a07f0ba1b0e5b4272b4a113b20ba"
server_key_iv: "2a3ba27227c9e10dd86dba70fdab97e7"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.251921 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:44.261823 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:44.264931 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:44.265146 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:44.265743 16061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:44.266621 16061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:44.266695 16061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.266777 16061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:44.266821 16061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.276088 16061 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:43213
I20260808 18:37:44.276110 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:43213 every 8 connection(s)
I20260808 18:37:44.277076 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:44.279064 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16061
I20260808 18:37:44.279211 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:44.279554 28655 external_mini_cluster.cc:1428] Setting key 36b88a55218b9acf9e0d01603b110a90
Aug 08 18:37:44 dist-test-slave-fd7l krb5kdc[15652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:44.292217 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.278879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:53789 (local address 127.27.251.254:45285)
0808 18:37:44.279027 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.279031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.280164 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 18:37:44.281975 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.281979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.282036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.282116 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:44.283420 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.283966 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.285627 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.285889 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.286038 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.288747 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:44.288751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:44.288753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:44.288789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:44.290064 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.290597 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.290601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.290603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.290693 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.291056 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.291059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.291061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.291272 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:44.291388 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.291906 (+ 518us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.292033 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:37:44.293258 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.279404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:45285 (local address 127.27.251.195:53789)
0808 18:37:44.280003 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.280064 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.281757 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.282405 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.282413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.282772 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:44.283250 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.283264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.284106 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.284110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.285364 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.285375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.286081 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.286086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.286293 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.286968 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:44.286988 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:44.288570 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:44.290201 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:44.290207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:44.290222 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:44.290482 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:44.290800 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:44.290803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:44.290805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:44.290949 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:44.291387 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:44.291394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:44.291658 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.291846 (+ 188us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.292177 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":154,"threads_started":1}
I20260808 18:37:44.294351 16200 heartbeater.cc:347] Connected to a master server at 127.27.251.254:45285
I20260808 18:37:44.294584 16200 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:44.295094 16200 heartbeater.cc:511] Master 127.27.251.254:45285 requested a full tablet report, sending...
I20260808 18:37:44.296133 15697 ts_manager.cc:194] Registered new tserver with Master: e5448eb1446c4af9815836436609e870 (127.27.251.195:43213)
I20260808 18:37:44.296747 15697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:53789
I20260808 18:37:44.305985 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:44.316376 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.308088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.27.251.254:45285)
0808 18:37:44.308304 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.308310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.308454 (+ 144us) server_negotiation.cc:407] Connection header received
0808 18:37:44.308652 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.308658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.308714 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.308802 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:44.309525 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.310061 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.310929 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.311161 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.311276 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.313043 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:44.313055 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:44.313058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:44.313082 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:44.314378 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.314906 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.314916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.314919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.314994 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:44.315334 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:44.315338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:44.315339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:44.315558 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:44.315728 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.316009 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.316103 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 18:37:44.319602 15697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:44.322957 15697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:44.338490 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.332068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.27.251.193:42723)
0808 18:37:44.332741 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.332748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.333085 (+ 337us) server_negotiation.cc:407] Connection header received
0808 18:37:44.333186 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.333192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.333408 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.333568 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:44.333888 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335118 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.336668 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.337491 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.337588 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.337723 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.338185 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.338320 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":87,"threads_started":1}
I20260808 18:37:44.338590 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.331970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:43213 (local address 127.0.0.1:54392)
0808 18:37:44.333322 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.333336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.333518 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.334113 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.334116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.334135 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:44.334227 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.334233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335539 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.335543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.336958 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.336970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.338095 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.338099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.338186 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.338207 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.338293 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.338418 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":77,"thread_start_us":92,"threads_started":1}
I20260808 18:37:44.338901 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.332693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:39989 (local address 127.0.0.1:39154)
0808 18:37:44.333816 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.333829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.333923 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.334431 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.334434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.334450 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:44.334534 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.334539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335478 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.335484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.336958 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.336970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.338319 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.338323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.338399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.338417 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.338500 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.338557 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":83,"thread_start_us":113,"threads_started":1}
I20260808 18:37:44.339035 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.331940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42723 (local address 127.0.0.1:54658)
0808 18:37:44.332743 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.332792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.332928 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.333568 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.333577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.333605 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:44.333749 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.333756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335261 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.335264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.336502 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.336534 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.337829 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.337833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.337958 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.338013 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.338760 (+ 747us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.338847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260808 18:37:44.338564 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.332281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.27.251.195:43213)
0808 18:37:44.332981 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.332992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.333427 (+ 435us) server_negotiation.cc:407] Connection header received
0808 18:37:44.333685 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.333691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.333971 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.334123 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:44.334347 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335422 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.337102 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.337949 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.338077 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.338217 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.338327 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.338415 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":96,"threads_started":1}
I20260808 18:37:44.339048 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.332835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.27.251.194:39989)
0808 18:37:44.333223 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.333234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.334010 (+ 776us) server_negotiation.cc:407] Connection header received
0808 18:37:44.334116 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.334121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.334313 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.334447 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:44.334616 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.335340 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.337130 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.337956 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.338117 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.338255 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.338674 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.338874 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":166,"threads_started":1}
I20260808 18:37:44.341954 15840 tablet_service.cc:1467] Processing CreateTablet for tablet 73a8cde733534b1190fcc3a6988d39cd (DEFAULT_TABLE table=test-table [id=273530b138ed40bba5b281b386ceb0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:44.342087 16129 tablet_service.cc:1467] Processing CreateTablet for tablet 73a8cde733534b1190fcc3a6988d39cd (DEFAULT_TABLE table=test-table [id=273530b138ed40bba5b281b386ceb0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:44.342088 15981 tablet_service.cc:1467] Processing CreateTablet for tablet 73a8cde733534b1190fcc3a6988d39cd (DEFAULT_TABLE table=test-table [id=273530b138ed40bba5b281b386ceb0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:44.343027 15840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a8cde733534b1190fcc3a6988d39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:44.343130 15981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a8cde733534b1190fcc3a6988d39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:44.343130 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a8cde733534b1190fcc3a6988d39cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:44.349608 16217 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: Bootstrap starting.
I20260808 18:37:44.350546 16218 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Bootstrap starting.
I20260808 18:37:44.352342 16217 tablet_bootstrap.cc:654] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:44.352686 16218 tablet_bootstrap.cc:654] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:44.353386 16217 log.cc:824] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:44.354033 16218 log.cc:824] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:44.355872 16217 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: No bootstrap required, opened a new log
I20260808 18:37:44.356094 16217 ts_tablet_manager.cc:1397] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 18:37:44.356202 16218 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: No bootstrap required, opened a new log
I20260808 18:37:44.356390 16218 ts_tablet_manager.cc:1397] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 18:37:44.359344 16218 raft_consensus.cc:348] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.359614 16218 raft_consensus.cc:374] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:44.359705 16218 raft_consensus.cc:729] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf364c37d8464080035f1b2ea5a3ee, State: Initialized, Role: FOLLOWER
I20260808 18:37:44.359581 16217 raft_consensus.cc:348] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.359829 16219 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Bootstrap starting.
I20260808 18:37:44.359870 16217 raft_consensus.cc:374] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:44.359927 16217 raft_consensus.cc:729] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68b3dc1c5074865af71898d3918fc74, State: Initialized, Role: FOLLOWER
I20260808 18:37:44.360230 16218 consensus_queue.cc:260] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.360397 16217 consensus_queue.cc:260] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.361122 15909 heartbeater.cc:503] Master 127.27.251.254:45285 was elected leader, sending a full tablet report...
I20260808 18:37:44.361333 16218 ts_tablet_manager.cc:1428] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260808 18:37:44.361413 16217 ts_tablet_manager.cc:1428] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:44.361689 16055 heartbeater.cc:503] Master 127.27.251.254:45285 was elected leader, sending a full tablet report...
I20260808 18:37:44.362450 16219 tablet_bootstrap.cc:654] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:44.363931 16219 log.cc:824] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:44.366926 16219 tablet_bootstrap.cc:492] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: No bootstrap required, opened a new log
I20260808 18:37:44.367138 16219 ts_tablet_manager.cc:1397] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260808 18:37:44.370199 16219 raft_consensus.cc:348] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.370421 16219 raft_consensus.cc:374] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:44.370507 16219 raft_consensus.cc:729] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5448eb1446c4af9815836436609e870, State: Initialized, Role: FOLLOWER
I20260808 18:37:44.370980 16219 consensus_queue.cc:260] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.371768 16200 heartbeater.cc:503] Master 127.27.251.254:45285 was elected leader, sending a full tablet report...
I20260808 18:37:44.372107 16219 ts_tablet_manager.cc:1428] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:44.375458 16223 raft_consensus.cc:482] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:44.375609 16223 raft_consensus.cc:504] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.376726 16223 leader_election.cc:288] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68b3dc1c5074865af71898d3918fc74 (127.27.251.193:42723), e5448eb1446c4af9815836436609e870 (127.27.251.195:43213)
I20260808 18:37:44.381671 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.376958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42723 (local address 127.27.251.194:42263)
0808 18:37:44.377179 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.377198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.377307 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.377663 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.377665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.377678 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:44.377858 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.377865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.378908 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.378912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.380468 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.380477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.381199 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.381203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.381346 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.381375 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.381451 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.381531 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":186}
I20260808 18:37:44.382197 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.377082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:42263 (local address 127.27.251.193:42723)
0808 18:37:44.377430 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.377434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.377449 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:37:44.377500 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.377503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.377558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.377641 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:44.377999 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.378771 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.380608 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.381059 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.381199 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.381257 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.381955 (+ 698us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.382056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260808 18:37:44.382678 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.377115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:38979 (local address 127.27.251.195:43213)
0808 18:37:44.377530 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.377535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.377548 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:37:44.377590 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.377595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.377647 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.377716 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:44.377990 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.378819 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.381316 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.381983 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.382091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.382136 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.382460 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.382551 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260808 18:37:44.382686 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.376964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:43213 (local address 127.27.251.194:38979)
0808 18:37:44.377370 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.377382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.377467 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.377733 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.377736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.377748 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:44.377875 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.377880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.378914 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.378916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.380466 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.380477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.382219 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.382223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.382330 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.382359 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.382452 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.382549 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":163,"thread_start_us":130,"threads_started":1}
I20260808 18:37:44.382758 15860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a8cde733534b1190fcc3a6988d39cd" candidate_uuid: "abaf364c37d8464080035f1b2ea5a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b3dc1c5074865af71898d3918fc74" is_pre_election: true
I20260808 18:37:44.383047 15860 raft_consensus.cc:2454] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abaf364c37d8464080035f1b2ea5a3ee in term 0.
I20260808 18:37:44.383577 15934 leader_election.cc:302] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a68b3dc1c5074865af71898d3918fc74, abaf364c37d8464080035f1b2ea5a3ee; no voters:
I20260808 18:37:44.383834 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a8cde733534b1190fcc3a6988d39cd" candidate_uuid: "abaf364c37d8464080035f1b2ea5a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5448eb1446c4af9815836436609e870" is_pre_election: true
I20260808 18:37:44.383876 16223 raft_consensus.cc:2781] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:44.383977 16223 raft_consensus.cc:482] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:44.384037 16223 raft_consensus.cc:3037] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:44.384133 16151 raft_consensus.cc:2454] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abaf364c37d8464080035f1b2ea5a3ee in term 0.
I20260808 18:37:44.385249 16223 raft_consensus.cc:504] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.385594 16223 leader_election.cc:288] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [CANDIDATE]: Term 1 election: Requested vote from peers a68b3dc1c5074865af71898d3918fc74 (127.27.251.193:42723), e5448eb1446c4af9815836436609e870 (127.27.251.195:43213)
I20260808 18:37:44.385974 15860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a8cde733534b1190fcc3a6988d39cd" candidate_uuid: "abaf364c37d8464080035f1b2ea5a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b3dc1c5074865af71898d3918fc74"
I20260808 18:37:44.386085 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a8cde733534b1190fcc3a6988d39cd" candidate_uuid: "abaf364c37d8464080035f1b2ea5a3ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5448eb1446c4af9815836436609e870"
I20260808 18:37:44.386145 15860 raft_consensus.cc:3037] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:44.386240 16151 raft_consensus.cc:3037] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:44.387256 15860 raft_consensus.cc:2454] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abaf364c37d8464080035f1b2ea5a3ee in term 1.
I20260808 18:37:44.387262 16151 raft_consensus.cc:2454] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abaf364c37d8464080035f1b2ea5a3ee in term 1.
I20260808 18:37:44.387593 15933 leader_election.cc:302] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abaf364c37d8464080035f1b2ea5a3ee, e5448eb1446c4af9815836436609e870; no voters:
I20260808 18:37:44.387773 16223 raft_consensus.cc:2781] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:44.388015 16223 raft_consensus.cc:686] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 1 LEADER]: Becoming Leader. State: Replica: abaf364c37d8464080035f1b2ea5a3ee, State: Running, Role: LEADER
I20260808 18:37:44.388358 16223 consensus_queue.cc:237] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } }
I20260808 18:37:44.391572 15695 catalog_manager.cc:5495] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee reported cstate change: term changed from 0 to 1, leader changed from <none> to abaf364c37d8464080035f1b2ea5a3ee (127.27.251.194). New cstate: current_term: 1 leader_uuid: "abaf364c37d8464080035f1b2ea5a3ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abaf364c37d8464080035f1b2ea5a3ee" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 39989 } health_report { overall_health: HEALTHY } } }
I20260808 18:37:44.404325 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.400460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.27.251.194:39989)
0808 18:37:44.400708 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.400712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.400727 (+ 15us) server_negotiation.cc:407] Connection header received
0808 18:37:44.400789 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.400791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.400841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.400928 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:44.401192 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.401814 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.402762 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.402971 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.403057 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.403380 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:44.403768 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:44.403880 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.404076 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.404160 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260808 18:37:44.410491 16151 raft_consensus.cc:1261] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 1 FOLLOWER]: Refusing update from remote peer abaf364c37d8464080035f1b2ea5a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:44.410493 15860 raft_consensus.cc:1261] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 1 FOLLOWER]: Refusing update from remote peer abaf364c37d8464080035f1b2ea5a3ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:44.411298 16231 consensus_queue.cc:1035] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5448eb1446c4af9815836436609e870" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:44.411478 16223 consensus_queue.cc:1035] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68b3dc1c5074865af71898d3918fc74" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:44.420854 16236 mvcc.cc:205] Tried to move back new op lower bound from 7316333627017928704 to 7316333626937565184. Current Snapshot: MvccSnapshot[applied={T|T < 7316333627017928704}]
I20260808 18:37:44.427815 15697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:44.428052 15697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:44.430636 15697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80c4c3cb48e744f495148ea338c7a48e: Sending DeleteTablet for 3 replicas of tablet 73a8cde733534b1190fcc3a6988d39cd
I20260808 18:37:44.431407 16129 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a8cde733534b1190fcc3a6988d39cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:44 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:54392
I20260808 18:37:44.431425 15840 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a8cde733534b1190fcc3a6988d39cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:44 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:54658
I20260808 18:37:44.431933 16244 tablet_replica.cc:330] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74: stopping tablet replica
I20260808 18:37:44.432328 16244 raft_consensus.cc:2229] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:44.432380 15981 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a8cde733534b1190fcc3a6988d39cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:44 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:39154
I20260808 18:37:44.432757 16244 raft_consensus.cc:2258] T 73a8cde733534b1190fcc3a6988d39cd P a68b3dc1c5074865af71898d3918fc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:44.432927 16246 tablet_replica.cc:330] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: stopping tablet replica
I20260808 18:37:44.433012 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15768
I20260808 18:37:44.433144 16245 tablet_replica.cc:330] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: stopping tablet replica
I20260808 18:37:44.433415 16245 raft_consensus.cc:2229] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:44.433681 16245 raft_consensus.cc:2258] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:44.435397 16245 ts_tablet_manager.cc:1905] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:44.435875 16246 raft_consensus.cc:2229] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:44.436211 16246 raft_consensus.cc:2258] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:44.437719 16246 ts_tablet_manager.cc:1905] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:44.438251 16245 ts_tablet_manager.cc:1918] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:44.438336 16245 log.cc:1196] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/73a8cde733534b1190fcc3a6988d39cd
I20260808 18:37:44.438671 16245 ts_tablet_manager.cc:1939] T 73a8cde733534b1190fcc3a6988d39cd P e5448eb1446c4af9815836436609e870: Deleting consensus metadata
I20260808 18:37:44.439908 15684 catalog_manager.cc:4841] TS e5448eb1446c4af9815836436609e870 (127.27.251.195:43213): tablet 73a8cde733534b1190fcc3a6988d39cd (table test-table [id=273530b138ed40bba5b281b386ceb0f2]) successfully deleted
I20260808 18:37:44.440492 16246 ts_tablet_manager.cc:1918] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:44.440630 16246 log.cc:1196] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/73a8cde733534b1190fcc3a6988d39cd
I20260808 18:37:44.440958 16246 ts_tablet_manager.cc:1939] T 73a8cde733534b1190fcc3a6988d39cd P abaf364c37d8464080035f1b2ea5a3ee: Deleting consensus metadata
I20260808 18:37:44.442811 15683 catalog_manager.cc:4841] TS abaf364c37d8464080035f1b2ea5a3ee (127.27.251.194:39989): tablet 73a8cde733534b1190fcc3a6988d39cd (table test-table [id=273530b138ed40bba5b281b386ceb0f2]) successfully deleted
W20260808 18:37:44.450407 15685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:42723 (error 108)
I20260808 18:37:44.451326 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15914
I20260808 18:37:44.451659 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.451280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42723 (local address 127.0.0.1:54668)
0808 18:37:44.451452 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.451538 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 18:37:44.453142 15685 catalog_manager.cc:4584] TS a68b3dc1c5074865af71898d3918fc74 (127.27.251.193:42723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73a8cde733534b1190fcc3a6988d39cd: Network error: Client connection negotiation failed: client connection to 127.27.251.193:42723: connect: Connection refused (error 111)
I20260808 18:37:44.459127 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16061
I20260808 18:37:44.466045 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 15666
2026-08-08T18:37:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T18:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:44Z Disabled control of system clock
I20260808 18:37:44.499929 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38515
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:44.616942 16251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:44.617215 16251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:44.617251 16251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:44.621284 16251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:44.621341 16251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:44.621357 16251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:44.621404 16251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:44.621447 16251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:44.626282 16251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:44.627382 16251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:44.628880 16251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:44.637589 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.637565 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.637554 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:44.638105 16251 server_base.cc:1034] running on GCE node
I20260808 18:37:44.638649 16251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:44.639750 16251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:44.640934 16251 hybrid_clock.cc:648] HybridClock initialized: now 1786214264640897 us; error 45 us; skew 500 ppm
I20260808 18:37:44.642804 16251 webserver.cc:460] Webserver started at http://127.27.251.254:46783/ using document root <none> and password file <none>
I20260808 18:37:44.643384 16251 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:44.643473 16251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:44.643702 16251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:44.645777 16251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "0a9a75f59448176c18b539f10cba478f"
server_key_iv: "00ae20639979bfabc9882ea7a4d7e6e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.646353 16251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "0a9a75f59448176c18b539f10cba478f"
server_key_iv: "00ae20639979bfabc9882ea7a4d7e6e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.650067 16251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:37:44.652562 16266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.653803 16251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:44.653944 16251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "0a9a75f59448176c18b539f10cba478f"
server_key_iv: "00ae20639979bfabc9882ea7a4d7e6e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.654062 16251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:44.675966 16251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:44.676715 16251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:44.676934 16251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:44.684089 16251 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:38515
I20260808 18:37:44.684095 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:38515 every 8 connection(s)
I20260808 18:37:44.685081 16251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:44.686563 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16251
I20260808 18:37:44.686720 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:44.687024 28655 external_mini_cluster.cc:1428] Setting key 20b05fdfbe623d46329f13db26906da5
I20260808 18:37:44.688699 16323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:44.694314 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: Bootstrap starting.
I20260808 18:37:44.696722 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.688469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.27.251.254:38515)
0808 18:37:44.689296 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.689313 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.689349 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:37:44.690069 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.690086 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.690472 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.690826 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:44.691251 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.692221 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.693181 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.693676 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.693877 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.694860 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:44.694865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:44.694878 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:44.695264 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:44.695319 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:44.695339 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:44.695421 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.695608 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.695909 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":157,"threads_started":1}
I20260808 18:37:44.696887 16323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:44.697623 16323 log.cc:824] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:44.699536 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: No bootstrap required, opened a new log
I20260808 18:37:44.702992 16323 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } }
I20260808 18:37:44.703239 16323 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:44.703333 16323 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8550f35dd1405d8f0ceebaf5e3c0cd, State: Initialized, Role: FOLLOWER
I20260808 18:37:44.703794 16323 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } }
I20260808 18:37:44.703949 16323 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:44.704036 16323 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:44.704136 16323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:44.705099 16323 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } }
I20260808 18:37:44.705441 16323 leader_election.cc:302] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8550f35dd1405d8f0ceebaf5e3c0cd; no voters:
I20260808 18:37:44.705749 16323 leader_election.cc:288] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:44.705870 16328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:44.706274 16328 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [term 1 LEADER]: Becoming Leader. State: Replica: aa8550f35dd1405d8f0ceebaf5e3c0cd, State: Running, Role: LEADER
I20260808 18:37:44.706594 16328 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } }
I20260808 18:37:44.706939 16323 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:44.708621 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8550f35dd1405d8f0ceebaf5e3c0cd. Latest consensus state: current_term: 1 leader_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } } }
I20260808 18:37:44.708665 16329 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8550f35dd1405d8f0ceebaf5e3c0cd" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38515 } } }
I20260808 18:37:44.708753 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:44.708791 16329 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:44.709157 16343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:44.711900 16343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:44.717680 16343 catalog_manager.cc:1347] Generated new cluster ID: 389d2db73e3a43f6b101d8abb6f05b5d
I20260808 18:37:44.717761 16343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:44.754846 16343 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:44.755929 16343 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:44.767115 16343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: Generated new TSK 0
I20260808 18:37:44.767966 16343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:44.775159 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--builtin_ntp_servers=127.27.251.212:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:44.894851 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:44.895125 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:44.895165 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:44.899258 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:44.899313 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:44.899415 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:44.904379 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:44.905496 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:44.906944 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:44.917176 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:44.917272 16350 server_base.cc:1034] running on GCE node
W20260808 18:37:44.917181 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:44.917280 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:44.917824 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:44.918471 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:44.919641 16350 hybrid_clock.cc:648] HybridClock initialized: now 1786214264919614 us; error 31 us; skew 500 ppm
I20260808 18:37:44.921438 16350 webserver.cc:460] Webserver started at http://127.27.251.193:33007/ using document root <none> and password file <none>
I20260808 18:37:44.921984 16350 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:44.922042 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:44.922226 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:44.924242 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "8983d4f41eb84345b8eaa1d897579660"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "3eaf75e66c178ebc3576160c85d439f8"
server_key_iv: "970f3496d2b3f64dc004ea434a2ae911"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.924839 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "8983d4f41eb84345b8eaa1d897579660"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "3eaf75e66c178ebc3576160c85d439f8"
server_key_iv: "970f3496d2b3f64dc004ea434a2ae911"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.928680 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:37:44.931329 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.932652 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:44.932770 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "8983d4f41eb84345b8eaa1d897579660"
format_stamp: "Formatted at 2026-08-08 18:37:44 on dist-test-slave-fd7l"
server_key: "3eaf75e66c178ebc3576160c85d439f8"
server_key_iv: "970f3496d2b3f64dc004ea434a2ae911"
server_key_version: "encryptionkey@0"
I20260808 18:37:44.932873 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:44.958979 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:44.959753 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:44.959967 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:44.960677 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:44.961721 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:44.961795 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.961874 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:44.961915 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:44.971320 16350 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:40695
I20260808 18:37:44.971346 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:40695 every 8 connection(s)
I20260808 18:37:44.972287 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:44.973456 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16350
I20260808 18:37:44.973594 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:44.973991 28655 external_mini_cluster.cc:1428] Setting key 14855fcc463da4961f5c3c26affe13d2
I20260808 18:37:44.976290 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--builtin_ntp_servers=127.27.251.212:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:44.983889 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.974524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:41685 (local address 127.27.251.254:38515)
0808 18:37:44.974679 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:44.974683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:44.976034 (+ 1351us) server_negotiation.cc:407] Connection header received
0808 18:37:44.978241 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:44.978247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:44.978319 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:44.978424 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:44.979359 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.980187 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.981497 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.981926 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.982060 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.983219 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:44.983224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:44.983229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:44.983248 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:44.983280 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:44.983293 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:44.983388 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:44.983566 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 18:37:44.983668 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:37:44.984763 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:44.975020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38515 (local address 127.27.251.193:41685)
0808 18:37:44.975867 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:44.975925 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:44.977975 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:44.978605 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:44.978612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:44.978700 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:44.979209 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.979222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.980321 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.980323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.981342 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:44.981361 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:44.982083 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:44.982086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:44.982257 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:44.982935 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:44.982936 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:44.982976 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:44.983043 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:44.983398 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:44.983407 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:44.983412 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:37:44.983520 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 18:37:44.983760 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260808 18:37:44.986128 16496 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38515
I20260808 18:37:44.986413 16496 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:44.987120 16496 heartbeater.cc:511] Master 127.27.251.254:38515 requested a full tablet report, sending...
I20260808 18:37:44.988952 16284 ts_manager.cc:194] Registered new tserver with Master: 8983d4f41eb84345b8eaa1d897579660 (127.27.251.193:40695)
I20260808 18:37:44.990190 16284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.193:41685
W20260808 18:37:45.094589 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:45.094852 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:45.094890 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:45.099030 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:45.099080 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:45.099134 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:45.103888 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:45.105062 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:45.106428 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:45.116410 16506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:45.116405 16508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:45.116405 16505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:45.116832 16499 server_base.cc:1034] running on GCE node
I20260808 18:37:45.117296 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:45.117916 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:45.119138 16499 hybrid_clock.cc:648] HybridClock initialized: now 1786214265119115 us; error 33 us; skew 500 ppm
I20260808 18:37:45.121285 16499 webserver.cc:460] Webserver started at http://127.27.251.194:36273/ using document root <none> and password file <none>
I20260808 18:37:45.121915 16499 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:45.121982 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:45.122277 16499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:45.124648 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "e0d1c57314ac41e9bda2163151b0408b"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "a435a84aa5451417f4fa74489de8f5cb"
server_key_iv: "ee41f56ece9c06cf2c78f88d2706b4f7"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.125272 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "e0d1c57314ac41e9bda2163151b0408b"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "a435a84aa5451417f4fa74489de8f5cb"
server_key_iv: "ee41f56ece9c06cf2c78f88d2706b4f7"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.129423 16499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:45.132057 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.133411 16499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:45.133517 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "e0d1c57314ac41e9bda2163151b0408b"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "a435a84aa5451417f4fa74489de8f5cb"
server_key_iv: "ee41f56ece9c06cf2c78f88d2706b4f7"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.133690 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:45.144902 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:45.145545 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:45.145700 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:45.146277 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:45.147217 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:45.147275 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.147316 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:45.147341 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.157514 16499 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:40407
I20260808 18:37:45.157580 16634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:40407 every 8 connection(s)
I20260808 18:37:45.158610 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:45.164425 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16499
I20260808 18:37:45.164597 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:45.164987 28655 external_mini_cluster.cc:1428] Setting key 8e1f82608f6f3e3dded05e62b7c2dfe1
I20260808 18:37:45.167410 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--builtin_ntp_servers=127.27.251.212:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:45.170063 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.160567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:45315 (local address 127.27.251.254:38515)
0808 18:37:45.160715 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.160719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.161864 (+ 1145us) server_negotiation.cc:407] Connection header received
0808 18:37:45.163501 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.163505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.163566 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.163689 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:45.164987 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.165555 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.167330 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.167643 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.167725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.169335 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:45.169338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:45.169341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:45.169365 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:45.169387 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:45.169396 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:45.169472 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.169748 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.169831 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 18:37:45.170950 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.161006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38515 (local address 127.27.251.194:45315)
0808 18:37:45.161704 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.161749 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.163269 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.163875 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.163882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.163976 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:45.164792 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.164819 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.165712 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.165715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.167185 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.167201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.167832 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.167840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.168099 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.168966 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:45.168971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:45.169037 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:45.169134 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:45.169511 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:45.169526 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:45.169535 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.169693 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.169972 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260808 18:37:45.172330 16635 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38515
I20260808 18:37:45.172665 16635 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:45.173486 16635 heartbeater.cc:511] Master 127.27.251.254:38515 requested a full tablet report, sending...
I20260808 18:37:45.174858 16284 ts_manager.cc:194] Registered new tserver with Master: e0d1c57314ac41e9bda2163151b0408b (127.27.251.194:40407)
I20260808 18:37:45.175490 16284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.194:45315
W20260808 18:37:45.293756 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:45.294103 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:45.294175 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:45.298183 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:45.298280 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:45.298373 16638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:45.303177 16638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:45.304384 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:45.305876 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:45.316442 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:45.316671 16638 server_base.cc:1034] running on GCE node
W20260808 18:37:45.316466 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:45.316442 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:45.317211 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:45.317927 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:45.319096 16638 hybrid_clock.cc:648] HybridClock initialized: now 1786214265319056 us; error 53 us; skew 500 ppm
I20260808 18:37:45.321276 16638 webserver.cc:460] Webserver started at http://127.27.251.195:41975/ using document root <none> and password file <none>
I20260808 18:37:45.321977 16638 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:45.322049 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:45.322342 16638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:45.324950 16638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "c175b2d55bc0440f85b5727a8bd1fc5d"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "69df8ab43a1b76980839f91b34c3022d"
server_key_iv: "bce8e4917cd878191d30006096031b19"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.325695 16638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "c175b2d55bc0440f85b5727a8bd1fc5d"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "69df8ab43a1b76980839f91b34c3022d"
server_key_iv: "bce8e4917cd878191d30006096031b19"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.330530 16638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:37:45.333666 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.335323 16638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:37:45.335484 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "c175b2d55bc0440f85b5727a8bd1fc5d"
format_stamp: "Formatted at 2026-08-08 18:37:45 on dist-test-slave-fd7l"
server_key: "69df8ab43a1b76980839f91b34c3022d"
server_key_iv: "bce8e4917cd878191d30006096031b19"
server_key_version: "encryptionkey@0"
I20260808 18:37:45.335633 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:45.366120 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:45.366940 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:45.367156 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:45.367859 16638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:45.368925 16638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:45.368975 16638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.369047 16638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:45.369081 16638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:45.378734 16638 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:42335
I20260808 18:37:45.378758 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:42335 every 8 connection(s)
I20260808 18:37:45.379776 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:45.387126 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16638
I20260808 18:37:45.387266 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:45.387586 28655 external_mini_cluster.cc:1428] Setting key 43f5a09e10315cb22213d3311ee92807
I20260808 18:37:45.390611 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.381796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:51447 (local address 127.27.251.254:38515)
0808 18:37:45.381971 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.381975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.383171 (+ 1196us) server_negotiation.cc:407] Connection header received
0808 18:37:45.384745 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.384748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.384801 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.384882 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:45.385934 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.386486 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.388223 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.388592 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.388714 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.390163 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:45.390167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:45.390169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:45.390181 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:45.390202 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:45.390210 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:45.390272 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.390436 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.390493 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 18:37:45.391660 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.382269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38515 (local address 127.27.251.195:51447)
0808 18:37:45.382983 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.383038 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.384382 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.385110 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.385118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.385236 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:45.385777 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.385793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.386634 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.386637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.388051 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.388088 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.388797 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.388801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.389029 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.389793 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:45.389795 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:45.389866 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:45.389969 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:45.390295 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:45.390303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:45.390308 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.390417 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.390769 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260808 18:37:45.392947 16774 heartbeater.cc:347] Connected to a master server at 127.27.251.254:38515
I20260808 18:37:45.393251 16774 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:45.393951 16774 heartbeater.cc:511] Master 127.27.251.254:38515 requested a full tablet report, sending...
I20260808 18:37:45.395117 16284 ts_manager.cc:194] Registered new tserver with Master: c175b2d55bc0440f85b5727a8bd1fc5d (127.27.251.195:42335)
I20260808 18:37:45.395702 16284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.195:51447
I20260808 18:37:45.402072 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:45.408105 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.404208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.27.251.254:38515)
0808 18:37:45.404362 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.404369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.404494 (+ 125us) server_negotiation.cc:407] Connection header received
0808 18:37:45.404660 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.404662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.404704 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.404769 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:45.404978 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.405480 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.406283 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.406589 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.406681 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.407486 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:45.407489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:45.407491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:45.407504 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:45.407524 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:45.407532 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:45.407738 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.407902 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.407940 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:37:45.411733 16284 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:45.415139 16284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:45.430847 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.424355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:42335 (local address 127.0.0.1:54270)
0808 18:37:45.424820 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.424869 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.424997 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.426317 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.426323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.426350 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:45.426523 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.426532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.428031 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.428034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.429315 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.429324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430279 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.430390 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430427 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.430543 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.430672 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":66,"thread_start_us":103,"threads_started":1}
I20260808 18:37:45.431015 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.424546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.27.251.195:42335)
0808 18:37:45.425206 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.425220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.425246 (+ 26us) server_negotiation.cc:407] Connection header received
0808 18:37:45.425330 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.425335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.426142 (+ 807us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.426317 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:45.426664 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.427891 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.429480 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430140 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430281 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430490 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.430573 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.430672 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":101,"threads_started":1}
I20260808 18:37:45.431139 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.425149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40695 (local address 127.0.0.1:57900)
0808 18:37:45.425766 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.425782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.425879 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.426585 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.426590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.426611 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:45.426689 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.426697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.428087 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.428091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.429289 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.429299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430590 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.430719 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430818 (+ 99us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.430893 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.430979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":94,"thread_start_us":83,"threads_started":1}
I20260808 18:37:45.431576 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.425056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.27.251.194:40407)
0808 18:37:45.425615 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.425628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.425895 (+ 267us) server_negotiation.cc:407] Connection header received
0808 18:37:45.425986 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.425991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.426188 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.426316 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:45.426712 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.427512 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.429464 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430227 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430353 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.431297 (+ 777us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.431409 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":164,"threads_started":1}
I20260808 18:37:45.431849 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.424409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:40407 (local address 127.0.0.1:60178)
0808 18:37:45.425687 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.425701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.425793 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.426469 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.426472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.426496 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:45.426577 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.426582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.427634 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.427639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.429279 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.429291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430590 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.430776 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430834 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.430893 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.430979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":222,"thread_start_us":140,"threads_started":1}
I20260808 18:37:45.432214 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.425444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.27.251.193:40695)
0808 18:37:45.425898 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.425911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.425962 (+ 51us) server_negotiation.cc:407] Connection header received
0808 18:37:45.426043 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.426047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.426285 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.426442 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:45.426817 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.427941 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.429416 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.430227 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.430353 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.430520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.431297 (+ 777us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.431408 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":148,"threads_started":1}
I20260808 18:37:45.434592 16421 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c (DEFAULT_TABLE table=test-table [id=5cb35b76cacc4067897ecc2ae97a1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:45.434592 16703 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c (DEFAULT_TABLE table=test-table [id=5cb35b76cacc4067897ecc2ae97a1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:45.434590 16564 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c (DEFAULT_TABLE table=test-table [id=5cb35b76cacc4067897ecc2ae97a1e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:45.435742 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:45.435742 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:45.435747 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:45.443455 16792 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Bootstrap starting.
I20260808 18:37:45.444861 16791 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Bootstrap starting.
I20260808 18:37:45.444890 16793 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Bootstrap starting.
I20260808 18:37:45.445848 16792 tablet_bootstrap.cc:654] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:45.447009 16793 tablet_bootstrap.cc:654] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:45.446990 16792 log.cc:824] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:45.447038 16791 tablet_bootstrap.cc:654] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:45.447808 16791 log.cc:824] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:45.447836 16793 log.cc:824] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:45.449750 16792 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: No bootstrap required, opened a new log
I20260808 18:37:45.449841 16793 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: No bootstrap required, opened a new log
I20260808 18:37:45.449841 16791 tablet_bootstrap.cc:492] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: No bootstrap required, opened a new log
I20260808 18:37:45.450016 16792 ts_tablet_manager.cc:1397] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 18:37:45.450042 16791 ts_tablet_manager.cc:1397] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:37:45.450042 16793 ts_tablet_manager.cc:1397] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 18:37:45.453143 16792 raft_consensus.cc:348] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.453645 16792 raft_consensus.cc:374] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:45.453763 16792 raft_consensus.cc:729] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d1c57314ac41e9bda2163151b0408b, State: Initialized, Role: FOLLOWER
I20260808 18:37:45.454358 16792 consensus_queue.cc:260] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.454581 16793 raft_consensus.cc:348] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.454928 16793 raft_consensus.cc:374] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:45.454860 16791 raft_consensus.cc:348] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.455122 16791 raft_consensus.cc:374] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:45.455142 16635 heartbeater.cc:503] Master 127.27.251.254:38515 was elected leader, sending a full tablet report...
I20260808 18:37:45.455175 16791 raft_consensus.cc:729] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8983d4f41eb84345b8eaa1d897579660, State: Initialized, Role: FOLLOWER
I20260808 18:37:45.455526 16792 ts_tablet_manager.cc:1428] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260808 18:37:45.455598 16793 raft_consensus.cc:729] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c175b2d55bc0440f85b5727a8bd1fc5d, State: Initialized, Role: FOLLOWER
I20260808 18:37:45.455662 16791 consensus_queue.cc:260] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.456074 16793 consensus_queue.cc:260] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.456739 16496 heartbeater.cc:503] Master 127.27.251.254:38515 was elected leader, sending a full tablet report...
I20260808 18:37:45.456854 16791 ts_tablet_manager.cc:1428] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 18:37:45.457208 16793 ts_tablet_manager.cc:1428] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 18:37:45.457578 16774 heartbeater.cc:503] Master 127.27.251.254:38515 was elected leader, sending a full tablet report...
W20260808 18:37:45.473613 16497 tablet.cc:2369] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:37:45.630875 16775 tablet.cc:2369] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:45.632905 16799 raft_consensus.cc:482] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:45.633085 16799 raft_consensus.cc:504] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.634207 16799 leader_election.cc:288] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c175b2d55bc0440f85b5727a8bd1fc5d (127.27.251.195:42335), 8983d4f41eb84345b8eaa1d897579660 (127.27.251.193:40695)
I20260808 18:37:45.638181 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.634558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:42335 (local address 127.27.251.194:36979)
0808 18:37:45.634733 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.634750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.634876 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.635301 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.635304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.635319 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:45.635400 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.635404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.636180 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.636183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.637068 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.637078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.637774 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.637778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.637870 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.637899 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.637981 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.638045 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 18:37:45.638244 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.634638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36979 (local address 127.27.251.195:42335)
0808 18:37:45.634780 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.634787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.634806 (+ 19us) server_negotiation.cc:407] Connection header received
0808 18:37:45.635043 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.635047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.635097 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.635194 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:45.635516 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.636020 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.637204 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.637640 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.637774 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.637826 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.638016 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.638094 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:37:45.638870 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8030adeb54b13a8f1fcbbb93a7a7c" candidate_uuid: "e0d1c57314ac41e9bda2163151b0408b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" is_pre_election: true
I20260808 18:37:45.638955 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.634861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40695 (local address 127.27.251.194:57109)
0808 18:37:45.635226 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.635242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:45.635375 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:45.635640 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:45.635643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:45.635660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:45.635787 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.635792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.636669 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.636674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.637653 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:45.637663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.638559 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.638565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:45.638655 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.638674 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 18:37:45.638763 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:37:45.638826 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260808 18:37:45.638967 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.635029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:57109 (local address 127.27.251.193:40695)
0808 18:37:45.635213 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.635217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.635322 (+ 105us) server_negotiation.cc:407] Connection header received
0808 18:37:45.635479 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.635482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.635537 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.635617 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:45.635896 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.636551 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.637777 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.638410 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.638535 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.638588 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.638784 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.638856 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 18:37:45.639230 16723 raft_consensus.cc:2454] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d1c57314ac41e9bda2163151b0408b in term 0.
I20260808 18:37:45.639657 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8030adeb54b13a8f1fcbbb93a7a7c" candidate_uuid: "e0d1c57314ac41e9bda2163151b0408b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983d4f41eb84345b8eaa1d897579660" is_pre_election: true
I20260808 18:37:45.639842 16518 leader_election.cc:302] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c175b2d55bc0440f85b5727a8bd1fc5d, e0d1c57314ac41e9bda2163151b0408b; no voters:
I20260808 18:37:45.639958 16443 raft_consensus.cc:2454] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d1c57314ac41e9bda2163151b0408b in term 0.
I20260808 18:37:45.640136 16799 raft_consensus.cc:2781] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:45.640256 16799 raft_consensus.cc:482] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:45.640331 16799 raft_consensus.cc:3037] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:45.641572 16799 raft_consensus.cc:504] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.641927 16799 leader_election.cc:288] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [CANDIDATE]: Term 1 election: Requested vote from peers c175b2d55bc0440f85b5727a8bd1fc5d (127.27.251.195:42335), 8983d4f41eb84345b8eaa1d897579660 (127.27.251.193:40695)
I20260808 18:37:45.642542 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8030adeb54b13a8f1fcbbb93a7a7c" candidate_uuid: "e0d1c57314ac41e9bda2163151b0408b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d"
I20260808 18:37:45.642567 16443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8030adeb54b13a8f1fcbbb93a7a7c" candidate_uuid: "e0d1c57314ac41e9bda2163151b0408b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8983d4f41eb84345b8eaa1d897579660"
I20260808 18:37:45.642684 16443 raft_consensus.cc:3037] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:45.642684 16723 raft_consensus.cc:3037] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:45.643987 16723 raft_consensus.cc:2454] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d1c57314ac41e9bda2163151b0408b in term 1.
I20260808 18:37:45.643980 16443 raft_consensus.cc:2454] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d1c57314ac41e9bda2163151b0408b in term 1.
I20260808 18:37:45.644286 16518 leader_election.cc:302] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c175b2d55bc0440f85b5727a8bd1fc5d, e0d1c57314ac41e9bda2163151b0408b; no voters:
I20260808 18:37:45.644588 16799 raft_consensus.cc:2781] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:45.644778 16799 raft_consensus.cc:686] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 1 LEADER]: Becoming Leader. State: Replica: e0d1c57314ac41e9bda2163151b0408b, State: Running, Role: LEADER
I20260808 18:37:45.645110 16799 consensus_queue.cc:237] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } }
I20260808 18:37:45.648553 16283 catalog_manager.cc:5495] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0d1c57314ac41e9bda2163151b0408b (127.27.251.194). New cstate: current_term: 1 leader_uuid: "e0d1c57314ac41e9bda2163151b0408b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0d1c57314ac41e9bda2163151b0408b" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 } health_report { overall_health: UNKNOWN } } }
W20260808 18:37:45.659888 16636 tablet.cc:2369] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:45.676709 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.671616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.27.251.194:40407)
0808 18:37:45.671783 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:45.671791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:45.671811 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:37:45.671946 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:45.671950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:45.672039 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:45.672181 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:45.672741 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.673673 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.674623 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:45.675048 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:45.675188 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:45.675550 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:45.676184 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:45.676360 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:45.676427 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 18:37:45.676476 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:37:45.684237 16443 raft_consensus.cc:1261] T c6d8030adeb54b13a8f1fcbbb93a7a7c P 8983d4f41eb84345b8eaa1d897579660 [term 1 FOLLOWER]: Refusing update from remote peer e0d1c57314ac41e9bda2163151b0408b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:45.684237 16723 raft_consensus.cc:1261] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 1 FOLLOWER]: Refusing update from remote peer e0d1c57314ac41e9bda2163151b0408b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:45.685173 16803 consensus_queue.cc:1035] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c175b2d55bc0440f85b5727a8bd1fc5d" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 42335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:45.685376 16799 consensus_queue.cc:1035] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8983d4f41eb84345b8eaa1d897579660" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 40695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:45.701035 16807 mvcc.cc:205] Tried to move back new op lower bound from 7316333632233857024 to 7316333632085098496. Current Snapshot: MvccSnapshot[applied={T|T < 7316333632233857024}]
I20260808 18:37:45.704486 16808 mvcc.cc:205] Tried to move back new op lower bound from 7316333632233857024 to 7316333632085098496. Current Snapshot: MvccSnapshot[applied={T|T < 7316333632233857024}]
I20260808 18:37:45.716946 16284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:45.717190 16284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:45.720780 16284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa8550f35dd1405d8f0ceebaf5e3c0cd: Sending DeleteTablet for 3 replicas of tablet c6d8030adeb54b13a8f1fcbbb93a7a7c
I20260808 18:37:45.722007 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:45 UTC) from {username='slave'} at 127.0.0.1:60178
I20260808 18:37:45.722138 16703 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:45 UTC) from {username='slave'} at 127.0.0.1:54270
I20260808 18:37:45.722788 16421 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:45 UTC) from {username='slave'} at 127.0.0.1:57900
I20260808 18:37:45.723284 16817 tablet_replica.cc:330] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: stopping tablet replica
I20260808 18:37:45.723656 16817 raft_consensus.cc:2229] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:45.723965 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16350
I20260808 18:37:45.724025 16817 raft_consensus.cc:2258] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:45.724444 16815 tablet_replica.cc:330] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: stopping tablet replica
I20260808 18:37:45.724814 16815 raft_consensus.cc:2229] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:45.725255 16815 raft_consensus.cc:2258] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:45.726792 16815 ts_tablet_manager.cc:1905] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:45.729943 16815 ts_tablet_manager.cc:1918] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:45.730050 16815 log.cc:1196] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/c6d8030adeb54b13a8f1fcbbb93a7a7c
I20260808 18:37:45.730947 16817 ts_tablet_manager.cc:1905] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:45.733964 16817 ts_tablet_manager.cc:1918] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:45.734184 16817 log.cc:1196] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/c6d8030adeb54b13a8f1fcbbb93a7a7c
I20260808 18:37:45.734609 16817 ts_tablet_manager.cc:1939] T c6d8030adeb54b13a8f1fcbbb93a7a7c P c175b2d55bc0440f85b5727a8bd1fc5d: Deleting consensus metadata
I20260808 18:37:45.734795 16815 ts_tablet_manager.cc:1939] T c6d8030adeb54b13a8f1fcbbb93a7a7c P e0d1c57314ac41e9bda2163151b0408b: Deleting consensus metadata
I20260808 18:37:45.736477 16270 catalog_manager.cc:4841] TS c175b2d55bc0440f85b5727a8bd1fc5d (127.27.251.195:42335): tablet c6d8030adeb54b13a8f1fcbbb93a7a7c (table test-table [id=5cb35b76cacc4067897ecc2ae97a1e46]) successfully deleted
I20260808 18:37:45.738098 16268 catalog_manager.cc:4841] TS e0d1c57314ac41e9bda2163151b0408b (127.27.251.194:40407): tablet c6d8030adeb54b13a8f1fcbbb93a7a7c (table test-table [id=5cb35b76cacc4067897ecc2ae97a1e46]) successfully deleted
I20260808 18:37:45.742431 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16499
W20260808 18:37:45.742547 16270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:40695 (error 108)
I20260808 18:37:45.747394 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:45.746929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:40695 (local address 127.0.0.1:57916)
0808 18:37:45.747156 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:45.747267 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:40695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 18:37:45.747646 16270 catalog_manager.cc:4584] TS 8983d4f41eb84345b8eaa1d897579660 (127.27.251.193:40695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d8030adeb54b13a8f1fcbbb93a7a7c: Network error: Client connection negotiation failed: client connection to 127.27.251.193:40695: connect: Connection refused (error 111)
I20260808 18:37:45.752455 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16638
I20260808 18:37:45.760632 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16251
2026-08-08T18:37:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1295 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:45 dist-test-slave-fd7l krb5kdc[16823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:45 dist-test-slave-fd7l krb5kdc[16823](info): set up 2 sockets
Aug 08 18:37:45 dist-test-slave-fd7l krb5kdc[16823](info): commencing operation
krb5kdc: starting...
W20260808 18:37:46.843096 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:46 dist-test-slave-fd7l krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:37:46.980391 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35087
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:47.093411 16836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:47.093739 16836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:47.093822 16836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:47.097864 16836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:47.097949 16836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:47.097988 16836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:47.098048 16836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:47.098104 16836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:47.103403 16836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:47.104482 16836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:47.105553 16836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:47.114399 16843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.114394 16846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.114492 16844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.114765 16836 server_base.cc:1034] running on GCE node
I20260808 18:37:47.115401 16836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:47.116607 16836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:47.117810 16836 hybrid_clock.cc:648] HybridClock initialized: now 1786214267117798 us; error 40 us; skew 500 ppm
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:47.121011 16836 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:37:47.121950 16836 webserver.cc:460] Webserver started at http://127.27.251.254:35109/ using document root <none> and password file <none>
I20260808 18:37:47.122553 16836 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:47.122679 16836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:47.122918 16836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:47.125026 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "a082f2b0336c40b893c1c96eb9b91caf"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "8eb3a375d4cb7edfdd8cfb3f632cc7a8"
server_key_iv: "ffc0d55b7dbacc7aa2d6fbe2fb3de1e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.125598 16836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "a082f2b0336c40b893c1c96eb9b91caf"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "8eb3a375d4cb7edfdd8cfb3f632cc7a8"
server_key_iv: "ffc0d55b7dbacc7aa2d6fbe2fb3de1e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.129459 16836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:47.131959 16853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.133284 16836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:37:47.133441 16836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a082f2b0336c40b893c1c96eb9b91caf"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "8eb3a375d4cb7edfdd8cfb3f632cc7a8"
server_key_iv: "ffc0d55b7dbacc7aa2d6fbe2fb3de1e2"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.133558 16836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:47.162369 16836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:47.165578 16836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:47.165802 16836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:47.173732 16836 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35087
I20260808 18:37:47.173760 16910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35087 every 8 connection(s)
I20260808 18:37:47.174731 16836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:47.177591 16911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:47.177922 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16836
I20260808 18:37:47.178066 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:47.178399 28655 external_mini_cluster.cc:1428] Setting key a499895ffee154f5f7a6d1154906ed82
I20260808 18:37:47.185276 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: Bootstrap starting.
I20260808 18:37:47.188201 16911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: Neither blocks nor log segments found. Creating new log.
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:47.189159 16911 log.cc:824] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:47.191754 16911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: No bootstrap required, opened a new log
I20260808 18:37:47.194653 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.179897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.27.251.254:35087)
0808 18:37:47.180319 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:47.180339 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:47.180383 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:37:47.181246 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:47.181272 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:47.181741 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:47.182135 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:47.182856 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.184421 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.185383 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.186077 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.186241 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.189127 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:47.189134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:47.189151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:47.189187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:47.191291 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.191804 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.191810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.191817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.191963 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.192255 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.192258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.192260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.192826 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:47.193021 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:47.193240 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 18:37:47.193560 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":106,"threads_started":1}
I20260808 18:37:47.196480 16911 raft_consensus.cc:348] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } }
I20260808 18:37:47.196782 16911 raft_consensus.cc:374] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:47.196929 16911 raft_consensus.cc:729] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a082f2b0336c40b893c1c96eb9b91caf, State: Initialized, Role: FOLLOWER
I20260808 18:37:47.197379 16911 consensus_queue.cc:260] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } }
I20260808 18:37:47.197506 16911 raft_consensus.cc:388] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:47.197597 16911 raft_consensus.cc:482] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:47.197719 16911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:47.198771 16911 raft_consensus.cc:504] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } }
I20260808 18:37:47.199143 16911 leader_election.cc:302] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a082f2b0336c40b893c1c96eb9b91caf; no voters:
I20260808 18:37:47.199503 16911 leader_election.cc:288] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:47.199570 16916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:47.199836 16916 raft_consensus.cc:686] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [term 1 LEADER]: Becoming Leader. State: Replica: a082f2b0336c40b893c1c96eb9b91caf, State: Running, Role: LEADER
I20260808 18:37:47.200233 16916 consensus_queue.cc:237] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } }
I20260808 18:37:47.200857 16911 sys_catalog.cc:564] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:47.202014 16917 sys_catalog.cc:455] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a082f2b0336c40b893c1c96eb9b91caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } } }
I20260808 18:37:47.202193 16917 sys_catalog.cc:458] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:47.202623 16926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:47.202600 16918 sys_catalog.cc:455] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a082f2b0336c40b893c1c96eb9b91caf. Latest consensus state: current_term: 1 leader_uuid: "a082f2b0336c40b893c1c96eb9b91caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a082f2b0336c40b893c1c96eb9b91caf" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35087 } } }
I20260808 18:37:47.203400 16918 sys_catalog.cc:458] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:47.205651 16926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:47.211601 16926 catalog_manager.cc:1347] Generated new cluster ID: 2d02b733a8f746d2b5b8e2d884f84dab
I20260808 18:37:47.211725 16926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:47.236197 16926 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:47.237459 16926 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:47.241272 16926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: Generated new TSK 0
I20260808 18:37:47.241984 16926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.251.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
Entry for principal kudu/127.27.251.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab.
I20260808 18:37:47.281932 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:47.402199 16941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:47.402560 16941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:47.402639 16941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:47.406817 16941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:47.406880 16941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:47.406984 16941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:47.412559 16941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.193.keytab
--principal=kudu/127.27.251.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:47.413748 16941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:47.414875 16941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:47.424823 16947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.424871 16946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.425103 16941 server_base.cc:1034] running on GCE node
W20260808 18:37:47.424824 16949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.425580 16941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:47.426281 16941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:47.427495 16941 hybrid_clock.cc:648] HybridClock initialized: now 1786214267427461 us; error 46 us; skew 500 ppm
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:47.430629 16941 init.cc:377] Logged in from keytab as kudu/127.27.251.193@KRBTEST.COM (short username kudu)
I20260808 18:37:47.431555 16941 webserver.cc:460] Webserver started at http://127.27.251.193:33657/ using document root <none> and password file <none>
I20260808 18:37:47.432148 16941 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:47.432245 16941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:47.432494 16941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:47.434700 16941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "5f3a5a4d433d4c0481d24010c1a9da26"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "b822d5403132f5268f33ed27e96f8703"
server_key_iv: "2bbe32e1bbd903f6a0e65d865f752c87"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.435335 16941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "5f3a5a4d433d4c0481d24010c1a9da26"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "b822d5403132f5268f33ed27e96f8703"
server_key_iv: "2bbe32e1bbd903f6a0e65d865f752c87"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.439357 16941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260808 18:37:47.442018 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.443236 16941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:47.443377 16941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "5f3a5a4d433d4c0481d24010c1a9da26"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "b822d5403132f5268f33ed27e96f8703"
server_key_iv: "2bbe32e1bbd903f6a0e65d865f752c87"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.443502 16941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:47.481686 16941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:47.484835 16941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:47.485064 16941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:47.485689 16941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:47.486546 16941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:47.486625 16941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.486721 16941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:47.486773 16941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.495707 16941 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:34127
I20260808 18:37:47.495745 17079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:34127 every 8 connection(s)
I20260808 18:37:47.496683 16941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:47.500031 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 16941
I20260808 18:37:47.500201 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:47.500636 28655 external_mini_cluster.cc:1428] Setting key 9208ff6a1b18df0ca519c70dc345ad29
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.193@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.194@KRBTEST.COM; defaulting to no policy
I20260808 18:37:47.512183 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.498537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:52537 (local address 127.27.251.254:35087)
0808 18:37:47.498689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:47.498693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:47.499552 (+ 859us) server_negotiation.cc:407] Connection header received
0808 18:37:47.501437 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:47.501519 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:47.501573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:47.501675 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:47.503359 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.503932 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.505414 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.505666 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.505776 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.508402 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:47.508406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:47.508408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:47.508444 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:47.509981 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.510510 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.510514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.510515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.510607 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.510965 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.510970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.510971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.511219 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:47.511352 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:47.511867 (+ 515us) server_negotiation.cc:299] Negotiation successful
0808 18:37:47.512002 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":55}
I20260808 18:37:47.513099 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.498906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35087 (local address 127.27.251.193:52537)
0808 18:37:47.499389 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:47.499435 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:47.501133 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:47.501834 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:47.501841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:47.502397 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:47.503186 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.503210 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.504072 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.504078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.505266 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.505277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.505835 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.505839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.506021 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.506661 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:47.506678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:47.508233 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:47.510104 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.510110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.510126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.510398 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.510711 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.510714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.510716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.510864 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.511364 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:47.511372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:47.511627 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 18:37:47.511822 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 18:37:47.512173 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.194@KRBTEST.COM" created.
I20260808 18:37:47.514501 17080 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35087
I20260808 18:37:47.514777 17080 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:47.515400 17080 heartbeater.cc:511] Master 127.27.251.254:35087 requested a full tablet report, sending...
I20260808 18:37:47.517401 16872 ts_manager.cc:194] Registered new tserver with Master: 5f3a5a4d433d4c0481d24010c1a9da26 (127.27.251.193:34127)
I20260808 18:37:47.519137 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.193@KRBTEST.COM'} at 127.27.251.193:52537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
Entry for principal kudu/127.27.251.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab.
I20260808 18:37:47.532433 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:47.647558 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:47.647907 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:47.647956 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:47.652907 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:47.652973 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:47.653035 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:47.658699 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.194.keytab
--principal=kudu/127.27.251.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:47.659857 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:47.660890 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:47.670784 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.670771 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.670775 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.671170 17085 server_base.cc:1034] running on GCE node
I20260808 18:37:47.671528 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:47.672194 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:47.673393 17085 hybrid_clock.cc:648] HybridClock initialized: now 1786214267673359 us; error 45 us; skew 500 ppm
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:47.676388 17085 init.cc:377] Logged in from keytab as kudu/127.27.251.194@KRBTEST.COM (short username kudu)
I20260808 18:37:47.677412 17085 webserver.cc:460] Webserver started at http://127.27.251.194:40513/ using document root <none> and password file <none>
I20260808 18:37:47.678020 17085 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:47.678117 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:47.678363 17085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:47.680490 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "5b75338c586f47a98eff3fdcb0f76588"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "eb719d4693522971de169e746946cd53"
server_key_iv: "fbb0b5e7400e8edb042fd4db7c9e099e"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.681161 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "5b75338c586f47a98eff3fdcb0f76588"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "eb719d4693522971de169e746946cd53"
server_key_iv: "fbb0b5e7400e8edb042fd4db7c9e099e"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.685420 17085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:47.687885 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.689425 17085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:37:47.689548 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "5b75338c586f47a98eff3fdcb0f76588"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "eb719d4693522971de169e746946cd53"
server_key_iv: "fbb0b5e7400e8edb042fd4db7c9e099e"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.689697 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:47.702584 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:47.705816 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:47.705969 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:47.706494 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:47.707283 17085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:47.707329 17085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.707370 17085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:47.707386 17085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.716480 17085 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:36615
I20260808 18:37:47.716558 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:36615 every 8 connection(s)
I20260808 18:37:47.717479 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:47.719939 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17085
I20260808 18:37:47.720114 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:47.720479 28655 external_mini_cluster.cc:1428] Setting key c15bb76cb978035bf43cb45e436ce779
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.194@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.195@KRBTEST.COM; defaulting to no policy
I20260808 18:37:47.733327 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.719344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:36115 (local address 127.27.251.254:35087)
0808 18:37:47.719479 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:47.719483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:47.720409 (+ 926us) server_negotiation.cc:407] Connection header received
0808 18:37:47.722052 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:47.722056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:47.722108 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:47.722212 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:47.723684 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.724563 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.726388 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.726598 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.726671 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.729680 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:47.729683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:47.729685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:47.729710 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:47.731133 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.731696 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.731698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.731700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.731776 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.732196 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.732201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.732203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.732389 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:47.732505 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:47.733038 (+ 533us) server_negotiation.cc:299] Negotiation successful
0808 18:37:47.733168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:37:47.734264 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.719676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35087 (local address 127.27.251.194:36115)
0808 18:37:47.720215 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:47.720278 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:47.721818 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:47.722455 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:47.722463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:47.722893 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:47.723510 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.723529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.724713 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.724719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.726251 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.726267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.726751 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.726758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.727005 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.727667 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:47.727694 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:47.729483 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:47.731259 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.731265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.731281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.731561 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.731872 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.731876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.731878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.732026 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.732504 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:47.732537 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:47.732797 (+ 260us) client_negotiation.cc:769] Sending connection context
0808 18:37:47.732992 (+ 195us) client_negotiation.cc:240] Negotiation successful
0808 18:37:47.733467 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.195@KRBTEST.COM" created.
I20260808 18:37:47.735560 17227 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35087
I20260808 18:37:47.735821 17227 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:47.736323 17227 heartbeater.cc:511] Master 127.27.251.254:35087 requested a full tablet report, sending...
I20260808 18:37:47.737479 16872 ts_manager.cc:194] Registered new tserver with Master: 5b75338c586f47a98eff3fdcb0f76588 (127.27.251.194:36615)
I20260808 18:37:47.738375 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.194@KRBTEST.COM'} at 127.27.251.194:36115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
Entry for principal kudu/127.27.251.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab.
I20260808 18:37:47.752835 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:47.871672 17232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:47.871933 17232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:47.871977 17232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:47.876152 17232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:47.876233 17232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:47.876293 17232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:47.881968 17232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.195.keytab
--principal=kudu/127.27.251.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:35087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:47.883138 17232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:47.884241 17232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:47.894722 17240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:47.894709 17242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.894896 17232 server_base.cc:1034] running on GCE node
W20260808 18:37:47.895141 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:47.895614 17232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:47.896333 17232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:47.897521 17232 hybrid_clock.cc:648] HybridClock initialized: now 1786214267897482 us; error 54 us; skew 500 ppm
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:47.900713 17232 init.cc:377] Logged in from keytab as kudu/127.27.251.195@KRBTEST.COM (short username kudu)
I20260808 18:37:47.901729 17232 webserver.cc:460] Webserver started at http://127.27.251.195:42233/ using document root <none> and password file <none>
I20260808 18:37:47.902405 17232 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:47.902469 17232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:47.902689 17232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:47.904898 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "d430db1be9fb4f81bb58399ae5dfa715"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "5d74d1489ee5e7b3e3e9963440a2a9f9"
server_key_iv: "ee3b7cf500699f2e964481347b80bbb1"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.905488 17232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "d430db1be9fb4f81bb58399ae5dfa715"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "5d74d1489ee5e7b3e3e9963440a2a9f9"
server_key_iv: "ee3b7cf500699f2e964481347b80bbb1"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.909524 17232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:37:47.912137 17250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.913498 17232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:47.913645 17232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "d430db1be9fb4f81bb58399ae5dfa715"
format_stamp: "Formatted at 2026-08-08 18:37:47 on dist-test-slave-fd7l"
server_key: "5d74d1489ee5e7b3e3e9963440a2a9f9"
server_key_iv: "ee3b7cf500699f2e964481347b80bbb1"
server_key_version: "encryptionkey@0"
I20260808 18:37:47.913763 17232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:47.946695 17232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:47.950287 17232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:47.950515 17232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:47.951172 17232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:47.952059 17232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:47.952140 17232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.952217 17232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:47.952275 17232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:47.963106 17232 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:46445
I20260808 18:37:47.963125 17375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:46445 every 8 connection(s)
I20260808 18:37:47.964095 17232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:47.970850 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17232
I20260808 18:37:47.971025 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:47.971335 28655 external_mini_cluster.cc:1428] Setting key 775efb62b4cfcd99c9c3bc1e6a8883d3
Aug 08 18:37:47 dist-test-slave-fd7l krb5kdc[16823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214267, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.195@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:37:47.979892 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.966097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:43981 (local address 127.27.251.254:35087)
0808 18:37:47.966270 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:47.966276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:47.967225 (+ 949us) server_negotiation.cc:407] Connection header received
0808 18:37:47.968636 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:47.968641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:47.968705 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:47.968782 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:47.969990 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.970568 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.972196 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.972437 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.972578 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.975315 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:47.975320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:47.975322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:47.975360 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:47.977156 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.977853 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.977859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.977861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.977981 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.978493 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.978496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.978497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.978733 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:47.978869 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:47.979507 (+ 638us) server_negotiation.cc:299] Negotiation successful
0808 18:37:47.979676 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 18:37:47.980950 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.966538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:35087 (local address 127.27.251.195:43981)
0808 18:37:47.967060 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:47.967108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:47.968358 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:47.968946 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:47.968960 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:47.969340 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:47.969849 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.969863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.970768 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.970771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.972008 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:47.972022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.972638 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.972642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:47.972828 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.973498 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:37:47.973516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:47.975124 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:47.977326 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.977335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.977365 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.977712 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.978122 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:37:47.978126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:37:47.978128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:37:47.978317 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:37:47.978869 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:47.978880 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:47.979206 (+ 326us) client_negotiation.cc:769] Sending connection context
0808 18:37:47.979443 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 18:37:47.979852 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":114,"threads_started":1}
I20260808 18:37:47.982267 17376 heartbeater.cc:347] Connected to a master server at 127.27.251.254:35087
I20260808 18:37:47.982642 17376 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:47.983253 17376 heartbeater.cc:511] Master 127.27.251.254:35087 requested a full tablet report, sending...
I20260808 18:37:47.984560 16872 ts_manager.cc:194] Registered new tserver with Master: d430db1be9fb4f81bb58399ae5dfa715 (127.27.251.195:46445)
I20260808 18:37:47.985296 16872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.251.195@KRBTEST.COM'} at 127.27.251.195:43981
I20260808 18:37:47.985988 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:47.995908 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:47.988066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.27.251.254:35087)
0808 18:37:47.988222 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:47.988227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:47.988412 (+ 185us) server_negotiation.cc:407] Connection header received
0808 18:37:47.988479 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:47.988482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:47.988609 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:47.988682 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:47.989381 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.989906 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.990807 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:47.991028 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:47.991126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:47.992635 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:47.992639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:47.992641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:47.992669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:47.994151 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.994603 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.994606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.994607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.994691 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:47.994969 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:47.994972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:47.994973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:47.995217 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:47.995334 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:47.995565 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 18:37:47.995713 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 18:37:47.999130 16871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:48.002506 16871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:48.018240 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.012292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.27.251.193:34127)
0808 18:37:48.012731 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.012746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.013100 (+ 354us) server_negotiation.cc:407] Connection header received
0808 18:37:48.013192 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.013197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.013403 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.013574 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:48.013821 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.014605 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.016620 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.017408 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.017531 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.017678 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.017955 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.018067 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":84,"threads_started":1}
I20260808 18:37:48.018294 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.011794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:34127 (local address 127.0.0.1:39238)
0808 18:37:48.012874 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.012888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.012994 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.013556 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.013559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.013579 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:48.013692 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.013702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.014730 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.014737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.016465 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.016477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.017537 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.017542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.017669 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.017718 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.017921 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.018147 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":271,"thread_start_us":111,"threads_started":1}
I20260808 18:37:48.018392 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.012460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.27.251.195:46445)
0808 18:37:48.012906 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.012921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.013167 (+ 246us) server_negotiation.cc:407] Connection header received
0808 18:37:48.013386 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.013392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.013582 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.013754 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:48.013963 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.014998 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.016440 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.017394 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.017531 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.017678 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.018147 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.018249 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260808 18:37:48.018755 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.012226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46445 (local address 127.0.0.1:54416)
0808 18:37:48.013076 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.013090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.013229 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.013732 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.013738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.013760 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:48.013868 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.013874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.015134 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.015139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.016312 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.016321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.017537 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.017635 (+ 98us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.017715 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.017732 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.017926 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.018625 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":550,"thread_start_us":98,"threads_started":1}
I20260808 18:37:48.019193 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.011420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:36615 (local address 127.0.0.1:47172)
0808 18:37:48.011797 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.011854 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.012004 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.012892 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.012898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.012921 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:48.013135 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.013144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.014833 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.014838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.016465 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.016477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.018764 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.018771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.018880 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.018914 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.019003 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.019078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":189,"thread_start_us":126,"threads_started":1}
I20260808 18:37:48.019279 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.011623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.27.251.194:36615)
0808 18:37:48.012062 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.012077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.012104 (+ 27us) server_negotiation.cc:407] Connection header received
0808 18:37:48.012192 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.012198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.012447 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.012667 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:48.013287 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.014696 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.017742 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.018624 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.018759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.018889 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.019029 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.019135 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":111,"threads_started":1}
I20260808 18:37:48.022063 17008 tablet_service.cc:1467] Processing CreateTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 (DEFAULT_TABLE table=test-table [id=a668d610c7ea4c498c2731ae9e262c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:48.022076 17305 tablet_service.cc:1467] Processing CreateTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 (DEFAULT_TABLE table=test-table [id=a668d610c7ea4c498c2731ae9e262c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:48.022076 17156 tablet_service.cc:1467] Processing CreateTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 (DEFAULT_TABLE table=test-table [id=a668d610c7ea4c498c2731ae9e262c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:48.023176 17305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9455cd67bdbd42e5bbc97fc41744ac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:48.023176 17156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9455cd67bdbd42e5bbc97fc41744ac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:48.023177 17008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9455cd67bdbd42e5bbc97fc41744ac36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:48.030054 17395 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Bootstrap starting.
I20260808 18:37:48.031751 17397 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Bootstrap starting.
I20260808 18:37:48.032660 17395 tablet_bootstrap.cc:654] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:48.032903 17396 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Bootstrap starting.
I20260808 18:37:48.033622 17395 log.cc:824] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:48.034036 17397 tablet_bootstrap.cc:654] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:48.034976 17396 tablet_bootstrap.cc:654] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:48.035274 17397 log.cc:824] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:48.036188 17396 log.cc:824] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:48.036427 17395 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: No bootstrap required, opened a new log
I20260808 18:37:48.036655 17395 ts_tablet_manager.cc:1397] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260808 18:37:48.037469 17397 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: No bootstrap required, opened a new log
I20260808 18:37:48.037671 17397 ts_tablet_manager.cc:1397] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260808 18:37:48.038120 17396 tablet_bootstrap.cc:492] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: No bootstrap required, opened a new log
I20260808 18:37:48.038311 17396 ts_tablet_manager.cc:1397] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260808 18:37:48.041282 17396 raft_consensus.cc:348] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.041601 17396 raft_consensus.cc:374] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:48.041661 17396 raft_consensus.cc:729] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b75338c586f47a98eff3fdcb0f76588, State: Initialized, Role: FOLLOWER
I20260808 18:37:48.041587 17395 raft_consensus.cc:348] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.041679 17397 raft_consensus.cc:348] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.041879 17397 raft_consensus.cc:374] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:48.041936 17397 raft_consensus.cc:729] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430db1be9fb4f81bb58399ae5dfa715, State: Initialized, Role: FOLLOWER
I20260808 18:37:48.041874 17395 raft_consensus.cc:374] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:48.041998 17395 raft_consensus.cc:729] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3a5a4d433d4c0481d24010c1a9da26, State: Initialized, Role: FOLLOWER
I20260808 18:37:48.042186 17396 consensus_queue.cc:260] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.042433 17395 consensus_queue.cc:260] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.042431 17397 consensus_queue.cc:260] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.042992 17227 heartbeater.cc:503] Master 127.27.251.254:35087 was elected leader, sending a full tablet report...
I20260808 18:37:48.043272 17396 ts_tablet_manager.cc:1428] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 18:37:48.043397 17397 ts_tablet_manager.cc:1428] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260808 18:37:48.043709 17376 heartbeater.cc:503] Master 127.27.251.254:35087 was elected leader, sending a full tablet report...
I20260808 18:37:48.044610 17395 ts_tablet_manager.cc:1428] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 18:37:48.044885 17080 heartbeater.cc:503] Master 127.27.251.254:35087 was elected leader, sending a full tablet report...
I20260808 18:37:48.198081 17401 raft_consensus.cc:482] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:48.198310 17401 raft_consensus.cc:504] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.199864 17401 leader_election.cc:288] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d430db1be9fb4f81bb58399ae5dfa715 (127.27.251.195:46445), 5f3a5a4d433d4c0481d24010c1a9da26 (127.27.251.193:34127)
I20260808 18:37:48.204108 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.200060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:46445 (local address 127.27.251.194:37643)
0808 18:37:48.200211 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.200230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.200327 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.200684 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.200687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.200710 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:48.200841 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.200847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.201695 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.201699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.202752 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.202762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.203664 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.203668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.203777 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.203803 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.203885 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.203956 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 18:37:48.204115 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.200218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:37643 (local address 127.27.251.195:46445)
0808 18:37:48.200362 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.200368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.200389 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:37:48.200452 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.200456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.200552 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.200635 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:48.201016 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.201587 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.202890 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.203545 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.203635 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.203677 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.203937 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.204015 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:37:48.204767 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.200130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:34127 (local address 127.27.251.194:38297)
0808 18:37:48.200687 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.200701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.200847 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.201145 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.201148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.201169 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:48.201299 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.201304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.202299 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.202302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.203349 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.203360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.204248 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.204252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.204367 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.204390 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.204531 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.204604 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":104,"threads_started":1}
I20260808 18:37:48.204778 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.200570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:38297 (local address 127.27.251.193:34127)
0808 18:37:48.200711 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.200717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.200775 (+ 58us) server_negotiation.cc:407] Connection header received
0808 18:37:48.200938 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.200943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.200994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.201085 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:48.201415 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.202174 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.203476 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.204092 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.204221 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.204275 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.204557 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.204666 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:37:48.205062 17324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9455cd67bdbd42e5bbc97fc41744ac36" candidate_uuid: "5b75338c586f47a98eff3fdcb0f76588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430db1be9fb4f81bb58399ae5dfa715" is_pre_election: true
I20260808 18:37:48.205447 17324 raft_consensus.cc:2454] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b75338c586f47a98eff3fdcb0f76588 in term 0.
I20260808 18:37:48.205497 17030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9455cd67bdbd42e5bbc97fc41744ac36" candidate_uuid: "5b75338c586f47a98eff3fdcb0f76588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" is_pre_election: true
I20260808 18:37:48.205782 17030 raft_consensus.cc:2454] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b75338c586f47a98eff3fdcb0f76588 in term 0.
I20260808 18:37:48.206058 17102 leader_election.cc:302] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75338c586f47a98eff3fdcb0f76588, d430db1be9fb4f81bb58399ae5dfa715; no voters:
I20260808 18:37:48.206358 17401 raft_consensus.cc:2781] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:48.206501 17401 raft_consensus.cc:482] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:48.206579 17401 raft_consensus.cc:3037] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:48.207645 17401 raft_consensus.cc:504] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.208012 17401 leader_election.cc:288] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [CANDIDATE]: Term 1 election: Requested vote from peers d430db1be9fb4f81bb58399ae5dfa715 (127.27.251.195:46445), 5f3a5a4d433d4c0481d24010c1a9da26 (127.27.251.193:34127)
I20260808 18:37:48.208356 17324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9455cd67bdbd42e5bbc97fc41744ac36" candidate_uuid: "5b75338c586f47a98eff3fdcb0f76588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430db1be9fb4f81bb58399ae5dfa715"
I20260808 18:37:48.208459 17030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9455cd67bdbd42e5bbc97fc41744ac36" candidate_uuid: "5b75338c586f47a98eff3fdcb0f76588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3a5a4d433d4c0481d24010c1a9da26"
I20260808 18:37:48.208540 17324 raft_consensus.cc:3037] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:48.208612 17030 raft_consensus.cc:3037] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:48.209636 17324 raft_consensus.cc:2454] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b75338c586f47a98eff3fdcb0f76588 in term 1.
I20260808 18:37:48.209726 17030 raft_consensus.cc:2454] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b75338c586f47a98eff3fdcb0f76588 in term 1.
I20260808 18:37:48.210059 17102 leader_election.cc:302] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75338c586f47a98eff3fdcb0f76588, d430db1be9fb4f81bb58399ae5dfa715; no voters:
I20260808 18:37:48.210280 17401 raft_consensus.cc:2781] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:48.210536 17401 raft_consensus.cc:686] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 1 LEADER]: Becoming Leader. State: Replica: 5b75338c586f47a98eff3fdcb0f76588, State: Running, Role: LEADER
I20260808 18:37:48.210906 17401 consensus_queue.cc:237] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } }
I20260808 18:37:48.214185 16872 catalog_manager.cc:5495] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b75338c586f47a98eff3fdcb0f76588 (127.27.251.194). New cstate: current_term: 1 leader_uuid: "5b75338c586f47a98eff3fdcb0f76588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75338c586f47a98eff3fdcb0f76588" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 36615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 } health_report { overall_health: UNKNOWN } } }
W20260808 18:37:48.215219 17377 tablet.cc:2369] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:37:48.218816 17228 tablet.cc:2369] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:37:48.248409 17081 tablet.cc:2369] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:48.255669 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.251937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.27.251.194:36615)
0808 18:37:48.252093 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.252100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.252117 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:37:48.252210 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.252212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.252270 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.252388 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:48.252678 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.253258 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.254154 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.254385 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.254502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.254814 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:48.255230 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:48.255368 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.255432 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.255492 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 18:37:48.261703 17324 raft_consensus.cc:1261] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 1 FOLLOWER]: Refusing update from remote peer 5b75338c586f47a98eff3fdcb0f76588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:48.261703 17030 raft_consensus.cc:1261] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 1 FOLLOWER]: Refusing update from remote peer 5b75338c586f47a98eff3fdcb0f76588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:48.262486 17405 consensus_queue.cc:1035] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3a5a4d433d4c0481d24010c1a9da26" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:48.262739 17401 consensus_queue.cc:1035] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d430db1be9fb4f81bb58399ae5dfa715" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:48.269237 17408 mvcc.cc:205] Tried to move back new op lower bound from 7316333642791981056 to 7316333642594873344. Current Snapshot: MvccSnapshot[applied={T|T < 7316333642791981056}]
I20260808 18:37:48.270839 17410 mvcc.cc:205] Tried to move back new op lower bound from 7316333642791981056 to 7316333642594873344. Current Snapshot: MvccSnapshot[applied={T|T < 7316333642791981056}]
I20260808 18:37:48.272017 17409 mvcc.cc:205] Tried to move back new op lower bound from 7316333642791981056 to 7316333642594873344. Current Snapshot: MvccSnapshot[applied={T|T < 7316333642791981056}]
I20260808 18:37:48.278609 16870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:48.278796 16870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:48.281189 16870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a082f2b0336c40b893c1c96eb9b91caf: Sending DeleteTablet for 3 replicas of tablet 9455cd67bdbd42e5bbc97fc41744ac36
I20260808 18:37:48.282047 17305 tablet_service.cc:1514] Processing DeleteTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:48 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:54416
I20260808 18:37:48.282052 17156 tablet_service.cc:1514] Processing DeleteTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:48 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:47172
I20260808 18:37:48.282091 17008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9455cd67bdbd42e5bbc97fc41744ac36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:48 UTC) from {username='kudu', principal='kudu/127.27.251.254@KRBTEST.COM'} at 127.0.0.1:39238
I20260808 18:37:48.282665 17419 tablet_replica.cc:330] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26: stopping tablet replica
I20260808 18:37:48.282784 17417 tablet_replica.cc:330] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: stopping tablet replica
I20260808 18:37:48.283036 17417 raft_consensus.cc:2229] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:48.283027 17419 raft_consensus.cc:2229] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:48.283346 17419 raft_consensus.cc:2258] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5f3a5a4d433d4c0481d24010c1a9da26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:48.283398 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16941
I20260808 18:37:48.283731 17418 tablet_replica.cc:330] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: stopping tablet replica
I20260808 18:37:48.283527 17417 raft_consensus.cc:2258] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:48.284044 17418 raft_consensus.cc:2229] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:48.284332 17418 raft_consensus.cc:2258] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:48.285768 17418 ts_tablet_manager.cc:1905] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:48.285768 17417 ts_tablet_manager.cc:1905] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:48.288738 17418 ts_tablet_manager.cc:1918] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:48.288848 17418 log.cc:1196] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/9455cd67bdbd42e5bbc97fc41744ac36
I20260808 18:37:48.289193 17417 ts_tablet_manager.cc:1918] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:48.289222 17418 ts_tablet_manager.cc:1939] T 9455cd67bdbd42e5bbc97fc41744ac36 P d430db1be9fb4f81bb58399ae5dfa715: Deleting consensus metadata
I20260808 18:37:48.289455 17417 log.cc:1196] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/9455cd67bdbd42e5bbc97fc41744ac36
I20260808 18:37:48.289834 17417 ts_tablet_manager.cc:1939] T 9455cd67bdbd42e5bbc97fc41744ac36 P 5b75338c586f47a98eff3fdcb0f76588: Deleting consensus metadata
I20260808 18:37:48.290547 16854 catalog_manager.cc:4841] TS d430db1be9fb4f81bb58399ae5dfa715 (127.27.251.195:46445): tablet 9455cd67bdbd42e5bbc97fc41744ac36 (table test-table [id=a668d610c7ea4c498c2731ae9e262c98]) successfully deleted
I20260808 18:37:48.291276 16854 catalog_manager.cc:4841] TS 5b75338c586f47a98eff3fdcb0f76588 (127.27.251.194:36615): tablet 9455cd67bdbd42e5bbc97fc41744ac36 (table test-table [id=a668d610c7ea4c498c2731ae9e262c98]) successfully deleted
W20260808 18:37:48.295581 16857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:34127 (error 108)
I20260808 18:37:48.296588 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17085
I20260808 18:37:48.296784 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.296427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:34127 (local address 127.0.0.1:39242)
0808 18:37:48.296619 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.296701 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:34127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 18:37:48.298036 16857 catalog_manager.cc:4584] TS 5f3a5a4d433d4c0481d24010c1a9da26 (127.27.251.193:34127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9455cd67bdbd42e5bbc97fc41744ac36: Network error: Client connection negotiation failed: client connection to 127.27.251.193:34127: connect: Connection refused (error 111)
I20260808 18:37:48.304330 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17232
I20260808 18:37:48.311453 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 16836
2026-08-08T18:37:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2550 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T18:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:48Z Disabled control of system clock
I20260808 18:37:48.345261 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:44969
--webserver_interface=127.27.251.254
--webserver_port=0
--builtin_ntp_servers=127.27.251.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:48.459108 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:48.459385 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:48.459458 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:48.463776 17426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:48.463840 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:48.463856 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:48.463899 17426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:48.463934 17426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:48.468773 17426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:44969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:48.469923 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:48.471379 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 18:37:48.480665 17426 server_base.cc:1034] running on GCE node
W20260808 18:37:48.480594 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:48.480600 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:48.480679 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:48.481560 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:48.482779 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:48.483948 17426 hybrid_clock.cc:648] HybridClock initialized: now 1786214268483930 us; error 31 us; skew 500 ppm
I20260808 18:37:48.485965 17426 webserver.cc:460] Webserver started at http://127.27.251.254:36125/ using document root <none> and password file <none>
I20260808 18:37:48.486605 17426 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:48.486670 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:48.486904 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:48.489172 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "1256e55e15f44696a6135b5ebc1403fb"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "e3233cc523c7991a42204efff13c731a"
server_key_iv: "43eb99c4efd6f9aded4880ea73574d19"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.489770 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "1256e55e15f44696a6135b5ebc1403fb"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "e3233cc523c7991a42204efff13c731a"
server_key_iv: "43eb99c4efd6f9aded4880ea73574d19"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.493775 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:48.496347 17441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:48.497871 17426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 18:37:48.498019 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "1256e55e15f44696a6135b5ebc1403fb"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "e3233cc523c7991a42204efff13c731a"
server_key_iv: "43eb99c4efd6f9aded4880ea73574d19"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.498152 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:48.527396 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:48.528242 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:48.528462 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:48.537000 17426 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:44969
I20260808 18:37:48.537060 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:44969 every 8 connection(s)
I20260808 18:37:48.538164 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:48.541615 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:48.542117 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17426
I20260808 18:37:48.542268 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:48.542651 28655 external_mini_cluster.cc:1428] Setting key c90916ef09edb330680a64d5db165930
I20260808 18:37:48.548216 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: Bootstrap starting.
I20260808 18:37:48.550688 17496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:48.551611 17496 log.cc:824] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:48.553378 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.544199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.27.251.254:44969)
0808 18:37:48.545106 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.545123 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.545167 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:37:48.545907 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.545926 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.546365 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.546829 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:48.547337 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.548716 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.549747 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.550250 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.550430 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.551453 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:48.551458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:48.551472 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:48.551930 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:48.552006 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:48.552034 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:48.552167 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.552274 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.552622 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":150,"threads_started":1}
I20260808 18:37:48.553938 17496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: No bootstrap required, opened a new log
I20260808 18:37:48.558172 17496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } }
I20260808 18:37:48.558444 17496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:48.558533 17496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e55e15f44696a6135b5ebc1403fb, State: Initialized, Role: FOLLOWER
I20260808 18:37:48.559015 17496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } }
I20260808 18:37:48.559175 17496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:48.559267 17496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:48.559369 17496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:48.560611 17496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } }
I20260808 18:37:48.560979 17496 leader_election.cc:302] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1256e55e15f44696a6135b5ebc1403fb; no voters:
I20260808 18:37:48.561359 17496 leader_election.cc:288] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:48.561489 17501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:48.561782 17501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [term 1 LEADER]: Becoming Leader. State: Replica: 1256e55e15f44696a6135b5ebc1403fb, State: Running, Role: LEADER
I20260808 18:37:48.562107 17501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } }
I20260808 18:37:48.562753 17496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:48.563911 17504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1256e55e15f44696a6135b5ebc1403fb. Latest consensus state: current_term: 1 leader_uuid: "1256e55e15f44696a6135b5ebc1403fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } } }
I20260808 18:37:48.564054 17504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:48.564335 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1256e55e15f44696a6135b5ebc1403fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256e55e15f44696a6135b5ebc1403fb" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 44969 } } }
I20260808 18:37:48.564432 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:48.564580 17511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:48.568028 17511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:48.574645 17511 catalog_manager.cc:1347] Generated new cluster ID: 8efcc33e28024fb98e781594e1d30908
I20260808 18:37:48.574759 17511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:48.581269 17511 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:48.582250 17511 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:48.590566 17511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: Generated new TSK 0
I20260808 18:37:48.591396 17511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:48.600638 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.193:0
--local_ip_for_outbound_sockets=127.27.251.193
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--builtin_ntp_servers=127.27.251.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 18:37:48.737239 17521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:48.737501 17521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:48.737548 17521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:48.741721 17521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:48.741781 17521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:48.741864 17521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.193
I20260808 18:37:48.746627 17521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.251.193
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.193
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:48.748039 17521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:48.750030 17521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:48.761941 17527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:48.761943 17526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:48.762082 17521 server_base.cc:1034] running on GCE node
W20260808 18:37:48.762291 17529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:48.762698 17521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:48.763379 17521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:48.764575 17521 hybrid_clock.cc:648] HybridClock initialized: now 1786214268764557 us; error 25 us; skew 500 ppm
I20260808 18:37:48.766407 17521 webserver.cc:460] Webserver started at http://127.27.251.193:41237/ using document root <none> and password file <none>
I20260808 18:37:48.766952 17521 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:48.767011 17521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:48.767181 17521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:48.769377 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data/instance:
uuid: "ab9c03cf3f854c889c51273376117125"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "3dc0e6703d7816a12a67d37096463858"
server_key_iv: "9a9c136ff51c898811a37a8afa514b81"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.769912 17521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance:
uuid: "ab9c03cf3f854c889c51273376117125"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "3dc0e6703d7816a12a67d37096463858"
server_key_iv: "9a9c136ff51c898811a37a8afa514b81"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.773672 17521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 18:37:48.776141 17535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:48.777534 17521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:48.777664 17521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal
uuid: "ab9c03cf3f854c889c51273376117125"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "3dc0e6703d7816a12a67d37096463858"
server_key_iv: "9a9c136ff51c898811a37a8afa514b81"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.777781 17521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:48.813884 17521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:48.814725 17521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:48.814953 17521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:48.815622 17521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:48.816738 17521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:48.816818 17521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:48.816895 17521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:48.816938 17521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:48.826297 17521 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.193:42911
I20260808 18:37:48.826327 17658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.193:42911 every 8 connection(s)
I20260808 18:37:48.827251 17521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/data/info.pb
I20260808 18:37:48.830487 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17521
I20260808 18:37:48.830628 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-0/wal/instance
I20260808 18:37:48.831025 28655 external_mini_cluster.cc:1428] Setting key 17eacc5a17523c8b004df95abc6c1272
I20260808 18:37:48.833411 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.194:0
--local_ip_for_outbound_sockets=127.27.251.194
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--builtin_ntp_servers=127.27.251.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:48.838735 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.829286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.193:33701 (local address 127.27.251.254:44969)
0808 18:37:48.829447 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:48.829452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:48.830780 (+ 1328us) server_negotiation.cc:407] Connection header received
0808 18:37:48.832653 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:48.832656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:48.832715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:48.832837 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:48.833927 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.834794 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.836387 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.836728 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.836846 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.838050 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:48.838053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:48.838056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:48.838071 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:48.838117 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:48.838137 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:48.838217 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:48.838539 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 18:37:48.838611 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 18:37:48.840132 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:48.829693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44969 (local address 127.27.251.193:33701)
0808 18:37:48.830601 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:48.830666 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:48.832338 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:48.833040 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:48.833048 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:48.833156 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:48.833769 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.833785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.834946 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.834952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.836218 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:48.836230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:48.836909 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:48.836913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:48.837088 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:48.837766 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:48.837768 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:48.837811 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:48.837881 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:48.838262 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:48.838277 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:48.838288 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 18:37:48.838495 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 18:37:48.838875 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20260808 18:37:48.841529 17659 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44969
I20260808 18:37:48.841825 17659 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:48.842602 17659 heartbeater.cc:511] Master 127.27.251.254:44969 requested a full tablet report, sending...
I20260808 18:37:48.844300 17459 ts_manager.cc:194] Registered new tserver with Master: ab9c03cf3f854c889c51273376117125 (127.27.251.193:42911)
I20260808 18:37:48.845505 17459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.193:33701
W20260808 18:37:48.953433 17662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:48.953735 17662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:48.953816 17662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:48.957937 17662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:48.958004 17662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:48.958076 17662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.194
I20260808 18:37:48.963814 17662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.251.194
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.194
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:48.965250 17662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:48.966642 17662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:48.976790 17670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:48.976769 17668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:48.976833 17667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:48.977113 17662 server_base.cc:1034] running on GCE node
I20260808 18:37:48.977596 17662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:48.978365 17662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:48.979560 17662 hybrid_clock.cc:648] HybridClock initialized: now 1786214268979537 us; error 33 us; skew 500 ppm
I20260808 18:37:48.981796 17662 webserver.cc:460] Webserver started at http://127.27.251.194:36611/ using document root <none> and password file <none>
I20260808 18:37:48.982496 17662 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:48.982591 17662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:48.982818 17662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:48.985100 17662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data/instance:
uuid: "1d888cee20654394ab6302722d9f92ba"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "734ad1c177a570712d9f1c7c0e76172b"
server_key_iv: "c921080d3873157502dc39b11d8945bf"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.985692 17662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance:
uuid: "1d888cee20654394ab6302722d9f92ba"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "734ad1c177a570712d9f1c7c0e76172b"
server_key_iv: "c921080d3873157502dc39b11d8945bf"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.989563 17662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:37:48.992091 17677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:48.993455 17662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:48.993598 17662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal
uuid: "1d888cee20654394ab6302722d9f92ba"
format_stamp: "Formatted at 2026-08-08 18:37:48 on dist-test-slave-fd7l"
server_key: "734ad1c177a570712d9f1c7c0e76172b"
server_key_iv: "c921080d3873157502dc39b11d8945bf"
server_key_version: "encryptionkey@0"
I20260808 18:37:48.993717 17662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:49.025445 17662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:49.026235 17662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:49.026458 17662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:49.027099 17662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:49.028275 17662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:49.028373 17662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:49.028525 17662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:49.028584 17662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260808 18:37:49.038662 17662 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.194:43649
I20260808 18:37:49.038692 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.194:43649 every 8 connection(s)
I20260808 18:37:49.039748 17662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/data/info.pb
I20260808 18:37:49.041649 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17662
I20260808 18:37:49.041796 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal/instance
I20260808 18:37:49.042155 28655 external_mini_cluster.cc:1428] Setting key 5960fbeb5d8f5a5b07b53656245c3d01
I20260808 18:37:49.044258 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.251.195:0
--local_ip_for_outbound_sockets=127.27.251.195
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--builtin_ntp_servers=127.27.251.212:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 18:37:49.050686 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.041842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.194:39317 (local address 127.27.251.254:44969)
0808 18:37:49.041986 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.041991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.043264 (+ 1273us) server_negotiation.cc:407] Connection header received
0808 18:37:49.044756 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.044762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.044826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.044923 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:49.045877 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.046573 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.048359 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.048654 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.048737 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.050182 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:49.050185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:49.050187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:49.050201 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:49.050220 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:49.050228 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:49.050283 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.050497 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.050550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:37:49.052021 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.042354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44969 (local address 127.27.251.194:39317)
0808 18:37:49.043098 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.043153 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.044548 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.045098 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.045105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.045192 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:49.045732 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.045745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.046712 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.046718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.048228 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.048243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.048811 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.048818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.049072 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.049833 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:49.049838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:49.049901 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:49.049997 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:49.050317 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:49.050330 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:49.050338 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.050493 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.050834 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":158,"threads_started":1}
I20260808 18:37:49.053429 17803 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44969
I20260808 18:37:49.053699 17803 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:49.054271 17803 heartbeater.cc:511] Master 127.27.251.254:44969 requested a full tablet report, sending...
I20260808 18:37:49.055457 17459 ts_manager.cc:194] Registered new tserver with Master: 1d888cee20654394ab6302722d9f92ba (127.27.251.194:43649)
I20260808 18:37:49.056084 17459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.194:39317
W20260808 18:37:49.162519 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:49.162832 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:49.162914 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:49.167316 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:49.167404 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:49.167500 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.251.195
I20260808 18:37:49.172232 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.251.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.251.195
--webserver_port=0
--tserver_master_addrs=127.27.251.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.251.195
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:49.173400 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:49.174793 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:49.184736 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:49.184747 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:49.184780 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:49.185155 17807 server_base.cc:1034] running on GCE node
I20260808 18:37:49.185629 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:49.186395 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:49.187624 17807 hybrid_clock.cc:648] HybridClock initialized: now 1786214269187610 us; error 25 us; skew 500 ppm
I20260808 18:37:49.189684 17807 webserver.cc:460] Webserver started at http://127.27.251.195:42453/ using document root <none> and password file <none>
I20260808 18:37:49.190352 17807 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:49.190461 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:49.190886 17807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:49.193351 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data/instance:
uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7"
format_stamp: "Formatted at 2026-08-08 18:37:49 on dist-test-slave-fd7l"
server_key: "6712403edd1d9f279fafe087243c1458"
server_key_iv: "72a203e6a2dd730de873b15c9e8433ca"
server_key_version: "encryptionkey@0"
I20260808 18:37:49.193986 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance:
uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7"
format_stamp: "Formatted at 2026-08-08 18:37:49 on dist-test-slave-fd7l"
server_key: "6712403edd1d9f279fafe087243c1458"
server_key_iv: "72a203e6a2dd730de873b15c9e8433ca"
server_key_version: "encryptionkey@0"
I20260808 18:37:49.198318 17807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:37:49.201294 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:49.203011 17807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:37:49.203183 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal
uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7"
format_stamp: "Formatted at 2026-08-08 18:37:49 on dist-test-slave-fd7l"
server_key: "6712403edd1d9f279fafe087243c1458"
server_key_iv: "72a203e6a2dd730de873b15c9e8433ca"
server_key_version: "encryptionkey@0"
I20260808 18:37:49.203296 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:49.219074 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:49.219800 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:49.220018 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:49.220644 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 18:37:49.221621 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 18:37:49.221670 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:49.221740 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 18:37:49.221781 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:49.230877 17807 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.195:40887
I20260808 18:37:49.230906 17944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.195:40887 every 8 connection(s)
I20260808 18:37:49.231810 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/data/info.pb
I20260808 18:37:49.232438 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 17807
I20260808 18:37:49.232589 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal/instance
I20260808 18:37:49.232900 28655 external_mini_cluster.cc:1428] Setting key 4d386a14f737b50db585caad0e163e72
I20260808 18:37:49.241531 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.233868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:44379 (local address 127.27.251.254:44969)
0808 18:37:49.234031 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.234035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.235388 (+ 1353us) server_negotiation.cc:407] Connection header received
0808 18:37:49.236877 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.236880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.236938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.237021 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:49.237877 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.238407 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.239656 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.239887 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.239976 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.241109 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:49.241112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:49.241114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:49.241126 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:49.241147 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:49.241156 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:49.241234 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.241352 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.241403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:37:49.242424 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.234466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:44969 (local address 127.27.251.195:44379)
0808 18:37:49.235227 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.235279 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.236658 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.237158 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.237164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.237246 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:37:49.237741 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.237754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.238558 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.238561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.239487 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.239496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.240029 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.240031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.240174 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.240840 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 18:37:49.240842 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:37:49.240883 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 18:37:49.240948 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:37:49.241235 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:37:49.241239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:37:49.241243 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.241327 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.241544 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260808 18:37:49.243592 17945 heartbeater.cc:347] Connected to a master server at 127.27.251.254:44969
I20260808 18:37:49.243845 17945 heartbeater.cc:464] Registering TS with master...
I20260808 18:37:49.244377 17945 heartbeater.cc:511] Master 127.27.251.254:44969 requested a full tablet report, sending...
I20260808 18:37:49.245568 17459 ts_manager.cc:194] Registered new tserver with Master: 13ddb0ea2a0b4eebaef5009a8ca393c7 (127.27.251.195:40887)
I20260808 18:37:49.246381 17459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.251.195:44379
I20260808 18:37:49.247164 28655 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 18:37:49.253305 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.249324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.27.251.254:44969)
0808 18:37:49.249450 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.249455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.249692 (+ 237us) server_negotiation.cc:407] Connection header received
0808 18:37:49.249844 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.249848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.249893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.249995 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:49.250271 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.250777 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.251631 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.251868 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.251986 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.252848 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:49.252852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:49.252854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:49.252870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:49.252894 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:49.252903 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:49.253007 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.253107 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.253163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:37:49.256376 17458 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 18:37:49.259508 17458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 18:37:49.275077 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.268733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42911 (local address 127.0.0.1:48994)
0808 18:37:49.269349 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.269366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.269502 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.270489 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.270497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.270529 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:49.270724 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.270732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.272261 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.272264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.273531 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.273541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.274523 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.274637 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.274680 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.274791 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.274909 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":167,"spinlock_wait_cycles":4736,"thread_start_us":97,"threads_started":1}
I20260808 18:37:49.275107 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.269490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.27.251.193:42911)
0808 18:37:49.269973 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.269998 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.270027 (+ 29us) server_negotiation.cc:407] Connection header received
0808 18:37:49.270110 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.270117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.270334 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.270495 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:49.270865 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.272098 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.273670 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.274378 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274507 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.274706 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.274822 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.274934 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":113,"threads_started":1}
I20260808 18:37:49.275128 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.268696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43649 (local address 127.0.0.1:37394)
0808 18:37:49.269317 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.269366 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.269502 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.270574 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.270576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.270588 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:49.270789 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.270798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.272222 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.272225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.273880 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.273891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.274802 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.274884 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.274905 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.274950 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.275013 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":329,"spinlock_wait_cycles":3840,"thread_start_us":74,"threads_started":1}
I20260808 18:37:49.275686 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.269021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.195:40887 (local address 127.0.0.1:56998)
0808 18:37:49.270214 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.270227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.270315 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.271021 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.271023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.271038 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:49.271109 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.271113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.272103 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.272110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.273880 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.273891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.275177 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.275184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.275374 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.275400 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.275487 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.275558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":188,"thread_start_us":98,"threads_started":1}
I20260808 18:37:49.276001 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.269485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.27.251.195:40887)
0808 18:37:49.270082 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.270093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.270412 (+ 319us) server_negotiation.cc:407] Connection header received
0808 18:37:49.270505 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.270509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.270680 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.270829 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:49.271252 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.271989 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274020 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.274810 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274929 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.275091 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.275705 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.275826 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":133,"threads_started":1}
I20260808 18:37:49.276851 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.268803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.27.251.194:43649)
0808 18:37:49.270062 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.270075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.270099 (+ 24us) server_negotiation.cc:407] Connection header received
0808 18:37:49.270197 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.270203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.270407 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.270690 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:49.270908 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.272098 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.274045 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.274661 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.276232 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.276414 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.276463 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.276701 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":128,"threads_started":1}
I20260808 18:37:49.278563 17588 tablet_service.cc:1467] Processing CreateTablet for tablet 1604977422224b2596853b8547d0be90 (DEFAULT_TABLE table=test-table [id=f75476d070f544b99701ddd94cdc2cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:49.278940 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 1604977422224b2596853b8547d0be90 (DEFAULT_TABLE table=test-table [id=f75476d070f544b99701ddd94cdc2cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:49.279238 17728 tablet_service.cc:1467] Processing CreateTablet for tablet 1604977422224b2596853b8547d0be90 (DEFAULT_TABLE table=test-table [id=f75476d070f544b99701ddd94cdc2cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 18:37:49.279812 17588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1604977422224b2596853b8547d0be90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:49.279925 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1604977422224b2596853b8547d0be90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:49.280224 17728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1604977422224b2596853b8547d0be90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:49.286599 17963 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Bootstrap starting.
I20260808 18:37:49.288081 17964 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Bootstrap starting.
I20260808 18:37:49.288846 17963 tablet_bootstrap.cc:654] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:49.288858 17965 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Bootstrap starting.
I20260808 18:37:49.289592 17963 log.cc:824] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:49.290304 17964 tablet_bootstrap.cc:654] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:49.290925 17965 tablet_bootstrap.cc:654] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:49.291175 17964 log.cc:824] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:49.291764 17963 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: No bootstrap required, opened a new log
I20260808 18:37:49.291752 17965 log.cc:824] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:49.291993 17963 ts_tablet_manager.cc:1397] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:37:49.293305 17964 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: No bootstrap required, opened a new log
I20260808 18:37:49.293489 17965 tablet_bootstrap.cc:492] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: No bootstrap required, opened a new log
I20260808 18:37:49.293497 17964 ts_tablet_manager.cc:1397] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 18:37:49.293689 17965 ts_tablet_manager.cc:1397] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 18:37:49.296600 17963 raft_consensus.cc:348] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.296885 17963 raft_consensus.cc:374] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:49.296936 17963 raft_consensus.cc:729] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d888cee20654394ab6302722d9f92ba, State: Initialized, Role: FOLLOWER
I20260808 18:37:49.296965 17965 raft_consensus.cc:348] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.297307 17965 raft_consensus.cc:374] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:49.297380 17965 raft_consensus.cc:729] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ddb0ea2a0b4eebaef5009a8ca393c7, State: Initialized, Role: FOLLOWER
I20260808 18:37:49.297398 17963 consensus_queue.cc:260] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.297919 17965 consensus_queue.cc:260] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.298484 17803 heartbeater.cc:503] Master 127.27.251.254:44969 was elected leader, sending a full tablet report...
I20260808 18:37:49.298521 17963 ts_tablet_manager.cc:1428] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260808 18:37:49.298931 17965 ts_tablet_manager.cc:1428] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 18:37:49.299289 17945 heartbeater.cc:503] Master 127.27.251.254:44969 was elected leader, sending a full tablet report...
I20260808 18:37:49.299216 17964 raft_consensus.cc:348] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.299454 17964 raft_consensus.cc:374] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:49.299503 17964 raft_consensus.cc:729] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9c03cf3f854c889c51273376117125, State: Initialized, Role: FOLLOWER
I20260808 18:37:49.299903 17964 consensus_queue.cc:260] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.300853 17659 heartbeater.cc:503] Master 127.27.251.254:44969 was elected leader, sending a full tablet report...
I20260808 18:37:49.300868 17964 ts_tablet_manager.cc:1428] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260808 18:37:49.328564 17660 tablet.cc:2369] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:37:49.482899 17946 tablet.cc:2369] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 18:37:49.541018 17804 tablet.cc:2369] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:37:49.640048 17970 raft_consensus.cc:482] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 18:37:49.640352 17970 raft_consensus.cc:504] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.642113 17970 leader_election.cc:288] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d888cee20654394ab6302722d9f92ba (127.27.251.194:43649), ab9c03cf3f854c889c51273376117125 (127.27.251.193:42911)
I20260808 18:37:49.645877 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.642394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:38633 (local address 127.27.251.194:43649)
0808 18:37:49.642601 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.642606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.642627 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:37:49.642796 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.642799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.642843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.642934 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:49.643257 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.643774 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.644869 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.645311 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.645440 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.645494 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.645685 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.645756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 18:37:49.645879 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.642308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.194:43649 (local address 127.27.251.195:38633)
0808 18:37:49.642535 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.642552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.642654 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.642950 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.642953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.642971 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:49.643112 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.643119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.643893 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.643897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.644738 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.644748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.645440 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.645444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.645548 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.645574 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.645667 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.645740 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":21,"spinlock_wait_cycles":18304}
I20260808 18:37:49.646586 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.642411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42911 (local address 127.27.251.195:50891)
0808 18:37:49.642838 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.642850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 18:37:49.643012 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:37:49.643310 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:37:49.643313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:37:49.643330 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 18:37:49.643470 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.643475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.644270 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.644274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.645288 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:37:49.645304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.646236 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.646239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:37:49.646338 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.646363 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 18:37:49.646431 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 18:37:49.646478 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":146,"threads_started":1}
I20260808 18:37:49.646700 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.642540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.251.195:50891 (local address 127.27.251.193:42911)
0808 18:37:49.642685 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.642692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.642950 (+ 258us) server_negotiation.cc:407] Connection header received
0808 18:37:49.643138 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.643142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.643199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.643296 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 18:37:49.643598 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.644139 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.645458 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.646101 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.646231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.646288 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.646487 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.646569 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:37:49.646683 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1604977422224b2596853b8547d0be90" candidate_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d888cee20654394ab6302722d9f92ba" is_pre_election: true
I20260808 18:37:49.647112 17750 raft_consensus.cc:2454] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ddb0ea2a0b4eebaef5009a8ca393c7 in term 0.
I20260808 18:37:49.647392 17609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1604977422224b2596853b8547d0be90" candidate_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9c03cf3f854c889c51273376117125" is_pre_election: true
I20260808 18:37:49.647706 17825 leader_election.cc:302] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13ddb0ea2a0b4eebaef5009a8ca393c7, 1d888cee20654394ab6302722d9f92ba; no voters:
I20260808 18:37:49.647743 17609 raft_consensus.cc:2454] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ddb0ea2a0b4eebaef5009a8ca393c7 in term 0.
I20260808 18:37:49.648001 17970 raft_consensus.cc:2781] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 18:37:49.648108 17970 raft_consensus.cc:482] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 18:37:49.648195 17970 raft_consensus.cc:3037] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:49.649519 17970 raft_consensus.cc:504] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.649859 17970 leader_election.cc:288] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [CANDIDATE]: Term 1 election: Requested vote from peers 1d888cee20654394ab6302722d9f92ba (127.27.251.194:43649), ab9c03cf3f854c889c51273376117125 (127.27.251.193:42911)
I20260808 18:37:49.650205 17750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1604977422224b2596853b8547d0be90" candidate_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d888cee20654394ab6302722d9f92ba"
I20260808 18:37:49.650249 17609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1604977422224b2596853b8547d0be90" candidate_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9c03cf3f854c889c51273376117125"
I20260808 18:37:49.650352 17750 raft_consensus.cc:3037] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:49.650363 17609 raft_consensus.cc:3037] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:49.651765 17750 raft_consensus.cc:2454] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ddb0ea2a0b4eebaef5009a8ca393c7 in term 1.
I20260808 18:37:49.651765 17609 raft_consensus.cc:2454] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ddb0ea2a0b4eebaef5009a8ca393c7 in term 1.
I20260808 18:37:49.652119 17826 leader_election.cc:302] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13ddb0ea2a0b4eebaef5009a8ca393c7, ab9c03cf3f854c889c51273376117125; no voters:
I20260808 18:37:49.652309 17970 raft_consensus.cc:2781] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:49.652633 17970 raft_consensus.cc:686] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 1 LEADER]: Becoming Leader. State: Replica: 13ddb0ea2a0b4eebaef5009a8ca393c7, State: Running, Role: LEADER
I20260808 18:37:49.653007 17970 consensus_queue.cc:237] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } }
I20260808 18:37:49.656366 17459 catalog_manager.cc:5495] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13ddb0ea2a0b4eebaef5009a8ca393c7 (127.27.251.195). New cstate: current_term: 1 leader_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13ddb0ea2a0b4eebaef5009a8ca393c7" member_type: VOTER last_known_addr { host: "127.27.251.195" port: 40887 } health_report { overall_health: HEALTHY } } }
I20260808 18:37:49.727698 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.724072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.27.251.195:40887)
0808 18:37:49.724227 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:49.724230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:49.724246 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:37:49.724371 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:49.724374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:49.724430 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:49.724545 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:37:49.724801 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.725311 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.726222 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:49.726428 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:49.726506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:49.726836 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:37:49.727273 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:37:49.727363 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:49.727476 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 18:37:49.727523 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 18:37:49.734123 17750 raft_consensus.cc:1261] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 1 FOLLOWER]: Refusing update from remote peer 13ddb0ea2a0b4eebaef5009a8ca393c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:49.734143 17609 raft_consensus.cc:1261] T 1604977422224b2596853b8547d0be90 P ab9c03cf3f854c889c51273376117125 [term 1 FOLLOWER]: Refusing update from remote peer 13ddb0ea2a0b4eebaef5009a8ca393c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:37:49.734840 17970 consensus_queue.cc:1035] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d888cee20654394ab6302722d9f92ba" member_type: VOTER last_known_addr { host: "127.27.251.194" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:49.735038 17973 consensus_queue.cc:1035] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9c03cf3f854c889c51273376117125" member_type: VOTER last_known_addr { host: "127.27.251.193" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:37:49.745744 17977 mvcc.cc:205] Tried to move back new op lower bound from 7316333648823164928 to 7316333648501747712. Current Snapshot: MvccSnapshot[applied={T|T < 7316333648823164928 or (T in {7316333648823164928})}]
I20260808 18:37:49.751425 17458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:49.751597 17458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 18:37:49.753928 17458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1256e55e15f44696a6135b5ebc1403fb: Sending DeleteTablet for 3 replicas of tablet 1604977422224b2596853b8547d0be90
I20260808 18:37:49.754725 17588 tablet_service.cc:1514] Processing DeleteTablet for tablet 1604977422224b2596853b8547d0be90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:49 UTC) from {username='slave'} at 127.0.0.1:48994
I20260808 18:37:49.754751 17728 tablet_service.cc:1514] Processing DeleteTablet for tablet 1604977422224b2596853b8547d0be90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:49 UTC) from {username='slave'} at 127.0.0.1:37394
I20260808 18:37:49.754863 17875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1604977422224b2596853b8547d0be90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 18:37:49 UTC) from {username='slave'} at 127.0.0.1:56998
I20260808 18:37:49.755270 17990 tablet_replica.cc:330] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: stopping tablet replica
I20260808 18:37:49.755582 17990 raft_consensus.cc:2229] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 18:37:49.755882 17988 tablet_replica.cc:330] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: stopping tablet replica
I20260808 18:37:49.755982 17990 raft_consensus.cc:2258] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:49.756026 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17521
I20260808 18:37:49.756177 17988 raft_consensus.cc:2229] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 18:37:49.756479 17988 raft_consensus.cc:2258] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 18:37:49.757689 17990 ts_tablet_manager.cc:1905] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:49.758050 17988 ts_tablet_manager.cc:1905] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 18:37:49.760689 17990 ts_tablet_manager.cc:1918] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:49.760684 17988 ts_tablet_manager.cc:1918] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 18:37:49.760779 17990 log.cc:1196] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-2/wal/wals/1604977422224b2596853b8547d0be90
I20260808 18:37:49.760779 17988 log.cc:1196] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Deleting WAL directory at /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786214007390461-28655-0/minicluster-data/ts-1/wal/wals/1604977422224b2596853b8547d0be90
I20260808 18:37:49.761101 17988 ts_tablet_manager.cc:1939] T 1604977422224b2596853b8547d0be90 P 1d888cee20654394ab6302722d9f92ba: Deleting consensus metadata
I20260808 18:37:49.761101 17990 ts_tablet_manager.cc:1939] T 1604977422224b2596853b8547d0be90 P 13ddb0ea2a0b4eebaef5009a8ca393c7: Deleting consensus metadata
I20260808 18:37:49.762265 17443 catalog_manager.cc:4841] TS 1d888cee20654394ab6302722d9f92ba (127.27.251.194:43649): tablet 1604977422224b2596853b8547d0be90 (table test-table [id=f75476d070f544b99701ddd94cdc2cfa]) successfully deleted
I20260808 18:37:49.762265 17442 catalog_manager.cc:4841] TS 13ddb0ea2a0b4eebaef5009a8ca393c7 (127.27.251.195:40887): tablet 1604977422224b2596853b8547d0be90 (table test-table [id=f75476d070f544b99701ddd94cdc2cfa]) successfully deleted
W20260808 18:37:49.768640 17444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.251.193:42911 (error 108)
I20260808 18:37:49.768772 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17662
I20260808 18:37:49.771201 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:49.770787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.193:42911 (local address 127.0.0.1:49010)
0808 18:37:49.770996 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 18:37:49.771074 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.193:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260808 18:37:49.771508 17444 catalog_manager.cc:4584] TS ab9c03cf3f854c889c51273376117125 (127.27.251.193:42911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1604977422224b2596853b8547d0be90: Network error: Client connection negotiation failed: client connection to 127.27.251.193:42911: connect: Connection refused (error 111)
I20260808 18:37:49.776850 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17807
I20260808 18:37:49.784271 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 17426
2026-08-08T18:37:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1471 ms)
[----------] 6 tests from EncryptionPolicyTest (11324 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:49 dist-test-slave-fd7l krb5kdc[17994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:49 dist-test-slave-fd7l krb5kdc[17994](info): set up 2 sockets
Aug 08 18:37:49 dist-test-slave-fd7l krb5kdc[17994](info): commencing operation
krb5kdc: starting...
W20260808 18:37:50.869277 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:50 dist-test-slave-fd7l krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:51.004055 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:51.121657 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:51.121951 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:51.122037 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:51.126608 18007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:51.126688 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:51.126724 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:51.126760 18007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:51.126796 18007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:51.133009 18007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:51.134153 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:51.135228 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:51.144068 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:51.144096 18013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:51.144253 18007 server_base.cc:1034] running on GCE node
W20260808 18:37:51.144081 18017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:51.145031 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:51.146203 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:51.147369 18007 hybrid_clock.cc:648] HybridClock initialized: now 1786214271147339 us; error 39 us; skew 500 ppm
Aug 08 18:37:51 dist-test-slave-fd7l krb5kdc[17994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:51.150193 18007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:51.151175 18007 webserver.cc:460] Webserver started at http://127.0.0.1:37939/ using document root <none> and password file <none>
I20260808 18:37:51.151741 18007 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:51.151834 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:51.152077 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:51.154166 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "8a36ac9b3c2e461fa72d01eecaf3596f"
format_stamp: "Formatted at 2026-08-08 18:37:51 on dist-test-slave-fd7l"
server_key: "f58d2f11707147884ecae6aeb949704b"
server_key_iv: "6a2798ebd0f98944863fdc098a7f5131"
server_key_version: "encryptionkey@0"
I20260808 18:37:51.154749 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "8a36ac9b3c2e461fa72d01eecaf3596f"
format_stamp: "Formatted at 2026-08-08 18:37:51 on dist-test-slave-fd7l"
server_key: "f58d2f11707147884ecae6aeb949704b"
server_key_iv: "6a2798ebd0f98944863fdc098a7f5131"
server_key_version: "encryptionkey@0"
I20260808 18:37:51.158502 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 18:37:51.160928 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:51.162248 18007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:51.162467 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "8a36ac9b3c2e461fa72d01eecaf3596f"
format_stamp: "Formatted at 2026-08-08 18:37:51 on dist-test-slave-fd7l"
server_key: "f58d2f11707147884ecae6aeb949704b"
server_key_iv: "6a2798ebd0f98944863fdc098a7f5131"
server_key_version: "encryptionkey@0"
I20260808 18:37:51.162596 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:51.182722 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:51.185745 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:51.185963 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:51.193557 18007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36759
I20260808 18:37:51.193584 18079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36759 every 8 connection(s)
I20260808 18:37:51.194553 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:51.197389 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:51.201941 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18007
I20260808 18:37:51.202121 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:51.202450 28655 external_mini_cluster.cc:1428] Setting key dfa7053b5a5b6da264e0cc8493635a61
I20260808 18:37:51.203359 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: Bootstrap starting.
I20260808 18:37:51.206080 18080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:51.206955 18080 log.cc:824] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:51.209293 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: No bootstrap required, opened a new log
Aug 08 18:37:51 dist-test-slave-fd7l krb5kdc[17994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:37:51.213937 18080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260808 18:37:51.214236 18080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:51.214309 18080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a36ac9b3c2e461fa72d01eecaf3596f, State: Initialized, Role: FOLLOWER
I20260808 18:37:51.214865 18080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260808 18:37:51.215049 18080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:51.215123 18080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:51.215260 18080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:51.216457 18080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260808 18:37:51.216825 18080 leader_election.cc:302] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a36ac9b3c2e461fa72d01eecaf3596f; no voters:
I20260808 18:37:51.217226 18080 leader_election.cc:288] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:51.217288 18085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:51.217473 18085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [term 1 LEADER]: Becoming Leader. State: Replica: 8a36ac9b3c2e461fa72d01eecaf3596f, State: Running, Role: LEADER
I20260808 18:37:51.217823 18085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } }
I20260808 18:37:51.218550 18080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:51.219810 18086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a36ac9b3c2e461fa72d01eecaf3596f. Latest consensus state: current_term: 1 leader_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260808 18:37:51.220328 18086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:51.220213 18087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a36ac9b3c2e461fa72d01eecaf3596f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36759 } } }
I20260808 18:37:51.220645 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:51.204004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47783 (local address 127.0.0.1:36759)
0808 18:37:51.204456 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:51.204478 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:51.204560 (+ 82us) server_negotiation.cc:407] Connection header received
0808 18:37:51.205384 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:51.205409 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:51.205850 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:51.206231 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:51.207254 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:51.208622 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:51.209731 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:51.210397 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:51.210558 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:51.214010 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:51.214016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:51.214033 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:51.214066 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:51.216342 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:51.217081 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:51.217088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:51.217097 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:51.217262 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:51.217777 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:51.217781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:51.217784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:51.218391 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:51.218734 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:51.219119 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 18:37:51.219381 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260808 18:37:51.220752 18087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f [sys.catalog]: This master's current role is: LEADER
W20260808 18:37:51.223678 18101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:37:51.223785 18101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:37:51.224165 18100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:51.224352 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:37:51.225169 18100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:51.232689 18100 catalog_manager.cc:1347] Generated new cluster ID: 32e6a3f8a7244ba49fc70f3e078d30fd
I20260808 18:37:51.232757 18100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:51.236980 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:51.227672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60939 (local address 127.0.0.1:36759)
0808 18:37:51.227772 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:51.227776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:51.228092 (+ 316us) server_negotiation.cc:407] Connection header received
0808 18:37:51.228199 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:51.228202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:51.228256 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:51.228350 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:51.229162 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:51.230545 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:51.231374 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:51.231766 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:51.231882 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:51.233618 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:51.233621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:51.233624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:51.233657 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:51.235123 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:51.235616 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:51.235622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:51.235628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:51.235701 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:51.236038 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:51.236042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:51.236043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:51.236219 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:51.236376 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:51.236734 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 18:37:51.236826 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:37:51.260339 18100 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:51.261420 18100 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:51.267462 18100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a36ac9b3c2e461fa72d01eecaf3596f: Generated new TSK 0
I20260808 18:37:51.268154 18100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:51.275770 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18007
W20260808 18:37:51.284873 17992 connection.cc:535] client connection to 127.0.0.1:36759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36759): Connection reset by peer (error 104)
2026-08-08T18:37:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:51 dist-test-slave-fd7l krb5kdc[18112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:51 dist-test-slave-fd7l krb5kdc[18112](info): set up 2 sockets
Aug 08 18:37:51 dist-test-slave-fd7l krb5kdc[18112](info): commencing operation
krb5kdc: starting...
W20260808 18:37:52.416923 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:52 dist-test-slave-fd7l krb5kdc[18112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:52.549520 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:52.663717 18127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:52.663978 18127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:52.664019 18127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:52.668138 18127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:52.668197 18127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:52.668216 18127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:52.668233 18127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:52.668248 18127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:52.674225 18127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:52.675346 18127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:52.676385 18127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:52.684942 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:52.685021 18132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:52.684938 18133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:52.685173 18127 server_base.cc:1034] running on GCE node
I20260808 18:37:52.685784 18127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:52.686935 18127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:52.688102 18127 hybrid_clock.cc:648] HybridClock initialized: now 1786214272688067 us; error 48 us; skew 500 ppm
Aug 08 18:37:52 dist-test-slave-fd7l krb5kdc[18112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:52.690960 18127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:52.691900 18127 webserver.cc:460] Webserver started at http://127.0.0.1:45653/ using document root <none> and password file <none>
I20260808 18:37:52.692466 18127 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:52.692589 18127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:52.692839 18127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:52.694875 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "49dcdc7ab6974f26b347ce280adaf69f"
format_stamp: "Formatted at 2026-08-08 18:37:52 on dist-test-slave-fd7l"
server_key: "0c9993dd019126fd4e540109b4b2b2d2"
server_key_iv: "145d6f332eab1abaa324e70dc18aa350"
server_key_version: "encryptionkey@0"
I20260808 18:37:52.695454 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "49dcdc7ab6974f26b347ce280adaf69f"
format_stamp: "Formatted at 2026-08-08 18:37:52 on dist-test-slave-fd7l"
server_key: "0c9993dd019126fd4e540109b4b2b2d2"
server_key_iv: "145d6f332eab1abaa324e70dc18aa350"
server_key_version: "encryptionkey@0"
I20260808 18:37:52.699373 18127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 18:37:52.701978 18142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:52.703223 18127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:37:52.703367 18127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "49dcdc7ab6974f26b347ce280adaf69f"
format_stamp: "Formatted at 2026-08-08 18:37:52 on dist-test-slave-fd7l"
server_key: "0c9993dd019126fd4e540109b4b2b2d2"
server_key_iv: "145d6f332eab1abaa324e70dc18aa350"
server_key_version: "encryptionkey@0"
I20260808 18:37:52.703482 18127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:52.714748 18127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:52.717800 18127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:52.718012 18127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:52.725543 18127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43135
I20260808 18:37:52.725570 18198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43135 every 8 connection(s)
I20260808 18:37:52.726537 18127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:52.729316 18199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:52.734577 18199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f: Bootstrap starting.
I20260808 18:37:52.736507 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18127
I20260808 18:37:52.736737 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:52.737037 28655 external_mini_cluster.cc:1428] Setting key 26b3b9f72bbb0cd7647e2b239e9898f8
I20260808 18:37:52.737048 18199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:52.737867 18199 log.cc:824] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:52.740284 18199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f: No bootstrap required, opened a new log
I20260808 18:37:52.744824 18199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260808 18:37:52.745074 18199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:52.745136 18199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49dcdc7ab6974f26b347ce280adaf69f, State: Initialized, Role: FOLLOWER
I20260808 18:37:52.745642 18199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260808 18:37:52.745769 18199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:52.745857 18199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:52.745985 18199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:52.747040 18199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260808 18:37:52.747388 18199 leader_election.cc:302] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49dcdc7ab6974f26b347ce280adaf69f; no voters:
I20260808 18:37:52.747661 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:52.738162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43453 (local address 127.0.0.1:43135)
0808 18:37:52.738596 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:52.738611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:52.738647 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:37:52.739574 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:52.739600 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:52.740121 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:52.740488 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:52.741074 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:52.742672 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:52.743580 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:52.744297 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:52.744468 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:52.745611 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:52.745616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:52.745630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:52.746162 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:52.746220 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:52.746244 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:52.746362 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:52.746632 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 18:37:52.746776 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260808 18:37:52.747730 18199 leader_election.cc:288] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:52.747802 18204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:52.748040 18204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [term 1 LEADER]: Becoming Leader. State: Replica: 49dcdc7ab6974f26b347ce280adaf69f, State: Running, Role: LEADER
I20260808 18:37:52.748404 18204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } }
I20260808 18:37:52.749142 18199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:52.749611 18204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49dcdc7ab6974f26b347ce280adaf69f. Latest consensus state: current_term: 1 leader_uuid: "49dcdc7ab6974f26b347ce280adaf69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } } }
I20260808 18:37:52.750016 18204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:52.750116 18206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49dcdc7ab6974f26b347ce280adaf69f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49dcdc7ab6974f26b347ce280adaf69f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43135 } } }
I20260808 18:37:52.750299 18206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:52.750450 18214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:52.751032 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:37:52.753940 18214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:52.760165 18214 catalog_manager.cc:1347] Generated new cluster ID: dc3f5f9bebf642ea953914f45534d64a
I20260808 18:37:52.760229 18214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:52.761193 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:52.753216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53295 (local address 127.0.0.1:43135)
0808 18:37:52.753318 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:52.753322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:52.756943 (+ 3621us) server_negotiation.cc:407] Connection header received
0808 18:37:52.757067 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:52.757070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:52.757136 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:52.757221 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:52.757618 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:52.758280 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:52.759233 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:52.759483 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:52.759592 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:52.760629 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:52.760635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:52.760639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:52.760656 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:52.760689 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:52.760707 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:52.760808 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:52.761019 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 18:37:52.761062 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 18:37:52.775458 18214 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:52.776389 18214 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:52.786139 18214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49dcdc7ab6974f26b347ce280adaf69f: Generated new TSK 0
I20260808 18:37:52.786860 18214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:52.793042 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18127
W20260808 18:37:52.800836 18110 connection.cc:535] client connection to 127.0.0.1:43135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43135): Connection reset by peer (error 104)
2026-08-08T18:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:52 dist-test-slave-fd7l krb5kdc[18232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:52 dist-test-slave-fd7l krb5kdc[18232](info): set up 2 sockets
Aug 08 18:37:52 dist-test-slave-fd7l krb5kdc[18232](info): commencing operation
krb5kdc: starting...
W20260808 18:37:53.889458 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:53 dist-test-slave-fd7l krb5kdc[18232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:54.028882 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:54.144881 18245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:54.145149 18245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:54.145190 18245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:54.149341 18245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:54.149405 18245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:54.149430 18245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:54.149446 18245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:54.149462 18245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:54.155009 18245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:54.156111 18245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:54.157228 18245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:54.166232 18250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:54.166225 18251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:54.166266 18254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:54.166608 18245 server_base.cc:1034] running on GCE node
I20260808 18:37:54.167276 18245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:54.168337 18245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:54.169533 18245 hybrid_clock.cc:648] HybridClock initialized: now 1786214274169512 us; error 39 us; skew 500 ppm
Aug 08 18:37:54 dist-test-slave-fd7l krb5kdc[18232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:54.177397 18245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:54.178355 18245 webserver.cc:460] Webserver started at http://127.0.0.1:40671/ using document root <none> and password file <none>
I20260808 18:37:54.178960 18245 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:54.179059 18245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:54.179318 18245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:54.181535 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "d52dac48618a405aaa04d2e7f8249127"
format_stamp: "Formatted at 2026-08-08 18:37:54 on dist-test-slave-fd7l"
server_key: "f7423cf225941bc65e83e49808b3d921"
server_key_iv: "be64d127849fed15fb1c08777e94a5f1"
server_key_version: "encryptionkey@0"
I20260808 18:37:54.182123 18245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "d52dac48618a405aaa04d2e7f8249127"
format_stamp: "Formatted at 2026-08-08 18:37:54 on dist-test-slave-fd7l"
server_key: "f7423cf225941bc65e83e49808b3d921"
server_key_iv: "be64d127849fed15fb1c08777e94a5f1"
server_key_version: "encryptionkey@0"
I20260808 18:37:54.186249 18245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:37:54.188902 18261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:54.190335 18245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 18:37:54.190497 18245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "d52dac48618a405aaa04d2e7f8249127"
format_stamp: "Formatted at 2026-08-08 18:37:54 on dist-test-slave-fd7l"
server_key: "f7423cf225941bc65e83e49808b3d921"
server_key_iv: "be64d127849fed15fb1c08777e94a5f1"
server_key_version: "encryptionkey@0"
I20260808 18:37:54.190620 18245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:54.195559 18245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:54.199045 18245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:54.199265 18245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:54.207258 18245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45213
I20260808 18:37:54.207286 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45213 every 8 connection(s)
I20260808 18:37:54.208283 18245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:54.211155 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:54.216349 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18245
I20260808 18:37:54.216516 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:54.216811 28655 external_mini_cluster.cc:1428] Setting key dd6816d80fbe31ec74a9ceb22299f30b
I20260808 18:37:54.217065 18318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127: Bootstrap starting.
I20260808 18:37:54.219867 18318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:54.220826 18318 log.cc:824] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:54.222844 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:54.218266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54415 (local address 127.0.0.1:45213)
0808 18:37:54.218737 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:54.218756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:54.218800 (+ 44us) server_negotiation.cc:407] Connection header received
0808 18:37:54.219631 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:54.219667 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:54.220142 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:54.220550 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:54.221405 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:54.221410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:54.221422 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:54.221433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:54.221487 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:54.221505 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:54.221606 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:54.221791 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 18:37:54.221948 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260808 18:37:54.223181 18318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127: No bootstrap required, opened a new log
I20260808 18:37:54.225651 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:37:54.227422 18318 raft_consensus.cc:348] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260808 18:37:54.227715 18318 raft_consensus.cc:374] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:54.227797 18318 raft_consensus.cc:729] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52dac48618a405aaa04d2e7f8249127, State: Initialized, Role: FOLLOWER
I20260808 18:37:54.228276 18318 consensus_queue.cc:260] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260808 18:37:54.228459 18318 raft_consensus.cc:388] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:54.228559 18318 raft_consensus.cc:482] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:54.228670 18318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:54.229785 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:54.228078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45293 (local address 127.0.0.1:45213)
0808 18:37:54.228211 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:54.228216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:54.228391 (+ 175us) server_negotiation.cc:407] Connection header received
0808 18:37:54.228549 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:54.228553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:54.228593 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:54.228683 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:54.229420 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:54.229424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:54.229426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:54.229438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:54.229465 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:54.229478 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:54.229568 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:54.229633 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 18:37:54.229673 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:37:54.229965 18318 raft_consensus.cc:504] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260808 18:37:54.230324 18318 leader_election.cc:302] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52dac48618a405aaa04d2e7f8249127; no voters:
I20260808 18:37:54.230660 18318 leader_election.cc:288] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:54.230758 18328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:54.231061 18328 raft_consensus.cc:686] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [term 1 LEADER]: Becoming Leader. State: Replica: d52dac48618a405aaa04d2e7f8249127, State: Running, Role: LEADER
I20260808 18:37:54.231372 18328 consensus_queue.cc:237] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } }
I20260808 18:37:54.231974 18318 sys_catalog.cc:564] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:54.233204 18330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d52dac48618a405aaa04d2e7f8249127. Latest consensus state: current_term: 1 leader_uuid: "d52dac48618a405aaa04d2e7f8249127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } } }
I20260808 18:37:54.233209 18329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d52dac48618a405aaa04d2e7f8249127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52dac48618a405aaa04d2e7f8249127" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45213 } } }
I20260808 18:37:54.233340 18329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:54.233337 18330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:54.233752 18337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:54.237011 18337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:54.242270 18337 catalog_manager.cc:1347] Generated new cluster ID: dc5dd138c31b4167aa1b1d923f66bdaa
I20260808 18:37:54.242345 18337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:54.269757 18337 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:54.270138 18337 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:54.275924 18337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d52dac48618a405aaa04d2e7f8249127: Generated new TSK 0
I20260808 18:37:54.276726 18337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:54.281147 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18245
2026-08-08T18:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:54 dist-test-slave-fd7l krb5kdc[18349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:54 dist-test-slave-fd7l krb5kdc[18349](info): set up 2 sockets
Aug 08 18:37:54 dist-test-slave-fd7l krb5kdc[18349](info): commencing operation
krb5kdc: starting...
W20260808 18:37:55.382783 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:55.520684 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:55.638259 18364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:55.638535 18364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:55.638621 18364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:55.642664 18364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:55.642726 18364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:55.642748 18364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:55.642768 18364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:55.642827 18364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:55.648043 18364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:55.649091 18364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:55.650130 18364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:55.658985 18371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:55.659066 18372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:55.658991 18374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:55.659319 18364 server_base.cc:1034] running on GCE node
I20260808 18:37:55.660012 18364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:55.661231 18364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:55.662439 18364 hybrid_clock.cc:648] HybridClock initialized: now 1786214275662414 us; error 47 us; skew 500 ppm
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:55.665339 18364 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:55.666303 18364 webserver.cc:460] Webserver started at http://127.0.0.1:43869/ using document root <none> and password file <none>
I20260808 18:37:55.666890 18364 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:55.666985 18364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:55.667228 18364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:55.669386 18364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "106862a99ddd4e7a9125ff9fca8191fb"
format_stamp: "Formatted at 2026-08-08 18:37:55 on dist-test-slave-fd7l"
server_key: "629aa6f4530edf7e5378877915cd6b39"
server_key_iv: "8e95286496f837d94b18669011d75dbd"
server_key_version: "encryptionkey@0"
I20260808 18:37:55.669981 18364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "106862a99ddd4e7a9125ff9fca8191fb"
format_stamp: "Formatted at 2026-08-08 18:37:55 on dist-test-slave-fd7l"
server_key: "629aa6f4530edf7e5378877915cd6b39"
server_key_iv: "8e95286496f837d94b18669011d75dbd"
server_key_version: "encryptionkey@0"
I20260808 18:37:55.673933 18364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:37:55.676420 18381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:55.677739 18364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:37:55.677886 18364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "106862a99ddd4e7a9125ff9fca8191fb"
format_stamp: "Formatted at 2026-08-08 18:37:55 on dist-test-slave-fd7l"
server_key: "629aa6f4530edf7e5378877915cd6b39"
server_key_iv: "8e95286496f837d94b18669011d75dbd"
server_key_version: "encryptionkey@0"
I20260808 18:37:55.678002 18364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:55.688069 18364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:55.691069 18364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:55.691293 18364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:55.698590 18364 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40679
I20260808 18:37:55.698629 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40679 every 8 connection(s)
I20260808 18:37:55.699568 18364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:55.702700 18436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:55.708287 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18364
I20260808 18:37:55.708485 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:55.708848 28655 external_mini_cluster.cc:1428] Setting key 48b08cde7924f5547952ad533fe74113
I20260808 18:37:55.709244 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: Bootstrap starting.
I20260808 18:37:55.711916 18436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:55.712934 18436 log.cc:824] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:55.715355 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: No bootstrap required, opened a new log
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:37:55.720404 18436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } }
I20260808 18:37:55.720752 18436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:55.720873 18436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106862a99ddd4e7a9125ff9fca8191fb, State: Initialized, Role: FOLLOWER
I20260808 18:37:55.721426 18436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } }
I20260808 18:37:55.721596 18436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:55.721696 18436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:55.721813 18436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:55.723068 18436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } }
I20260808 18:37:55.723449 18436 leader_election.cc:302] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106862a99ddd4e7a9125ff9fca8191fb; no voters:
I20260808 18:37:55.723836 18436 leader_election.cc:288] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:55.723953 18441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:55.724195 18441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [term 1 LEADER]: Becoming Leader. State: Replica: 106862a99ddd4e7a9125ff9fca8191fb, State: Running, Role: LEADER
I20260808 18:37:55.724566 18441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } }
I20260808 18:37:55.725248 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:55.710591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50205 (local address 127.0.0.1:40679)
0808 18:37:55.711078 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:55.711094 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:55.711133 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:37:55.711806 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:55.711823 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:55.712230 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:55.712588 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:55.713627 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:55.714841 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:55.716101 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:55.716637 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:55.716764 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:55.719731 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:55.719736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:55.719749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:55.719777 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:55.721752 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:55.722312 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:55.722317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:55.722321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:55.722434 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:55.722739 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:55.722741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:55.722742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:55.723117 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:55.723339 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:55.723666 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 18:37:55.723989 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":148,"threads_started":1}
I20260808 18:37:55.725304 18436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:55.726007 18441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106862a99ddd4e7a9125ff9fca8191fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } } }
I20260808 18:37:55.726163 18441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:55.726507 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106862a99ddd4e7a9125ff9fca8191fb. Latest consensus state: current_term: 1 leader_uuid: "106862a99ddd4e7a9125ff9fca8191fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106862a99ddd4e7a9125ff9fca8191fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40679 } } }
I20260808 18:37:55.726627 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:55.729043 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 18:37:55.731151 18458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:37:55.731295 18458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:37:55.731447 18448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:55.732455 18448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:55.739369 18448 catalog_manager.cc:1347] Generated new cluster ID: 44a8769bf7d140c8a1050f989da45776
I20260808 18:37:55.739466 18448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:55.741690 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:55.731611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46773 (local address 127.0.0.1:40679)
0808 18:37:55.731723 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:55.731727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:55.731957 (+ 230us) server_negotiation.cc:407] Connection header received
0808 18:37:55.732068 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:55.732072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:55.732122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:55.732218 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:55.733228 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:55.734039 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:55.735338 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:55.735728 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:55.735847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:55.737801 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:55.737805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:55.737808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:37:55.737835 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:55.739803 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:55.740286 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:55.740290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:55.740292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:55.740383 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:37:55.740747 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:37:55.740751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:37:55.740753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:37:55.740957 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:55.741107 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:55.741379 (+ 272us) server_negotiation.cc:299] Negotiation successful
0808 18:37:55.741511 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 18:37:55.749925 18448 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:55.751215 18448 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:55.758901 18448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106862a99ddd4e7a9125ff9fca8191fb: Generated new TSK 0
I20260808 18:37:55.759647 18448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:55.764619 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18364
2026-08-08T18:37:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18470](info): set up 2 sockets
Aug 08 18:37:55 dist-test-slave-fd7l krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260808 18:37:56.857280 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:56 dist-test-slave-fd7l krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:56.992673 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:57.105219 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:57.105522 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:57.105620 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:57.109571 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:57.109658 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:57.109694 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:57.109731 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:57.109763 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:57.114900 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:57.115918 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:57.116951 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:57.125618 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:57.125591 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:57.125658 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:57.125818 18483 server_base.cc:1034] running on GCE node
I20260808 18:37:57.126374 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:57.127497 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:57.128688 18483 hybrid_clock.cc:648] HybridClock initialized: now 1786214277128667 us; error 53 us; skew 500 ppm
Aug 08 18:37:57 dist-test-slave-fd7l krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:57.131599 18483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:57.132498 18483 webserver.cc:460] Webserver started at http://127.0.0.1:33735/ using document root <none> and password file <none>
I20260808 18:37:57.133162 18483 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:57.133253 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:57.133489 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:57.135545 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "7e06998401814d1f875954db3b176e34"
format_stamp: "Formatted at 2026-08-08 18:37:57 on dist-test-slave-fd7l"
server_key: "4752129e55b2fe4c30c7f5077e7047ed"
server_key_iv: "72df07e23ada8b43f6968a3f7aa77825"
server_key_version: "encryptionkey@0"
I20260808 18:37:57.136123 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "7e06998401814d1f875954db3b176e34"
format_stamp: "Formatted at 2026-08-08 18:37:57 on dist-test-slave-fd7l"
server_key: "4752129e55b2fe4c30c7f5077e7047ed"
server_key_iv: "72df07e23ada8b43f6968a3f7aa77825"
server_key_version: "encryptionkey@0"
I20260808 18:37:57.139883 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260808 18:37:57.142346 18499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:57.143599 18483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:37:57.143760 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "7e06998401814d1f875954db3b176e34"
format_stamp: "Formatted at 2026-08-08 18:37:57 on dist-test-slave-fd7l"
server_key: "4752129e55b2fe4c30c7f5077e7047ed"
server_key_iv: "72df07e23ada8b43f6968a3f7aa77825"
server_key_version: "encryptionkey@0"
I20260808 18:37:57.143872 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:57.165539 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:57.169077 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:57.169299 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:57.176981 18483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44713
I20260808 18:37:57.177048 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44713 every 8 connection(s)
I20260808 18:37:57.178000 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:57.180727 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18483
I20260808 18:37:57.180898 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:57.181093 18556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:57.181223 28655 external_mini_cluster.cc:1428] Setting key 6d7838b47f98d4661aeddf2d545a6dc7
I20260808 18:37:57.189229 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34: Bootstrap starting.
I20260808 18:37:57.192030 18556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:57.192169 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:57.182710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35395 (local address 127.0.0.1:44713)
0808 18:37:57.183236 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:57.183258 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:57.183305 (+ 47us) server_negotiation.cc:407] Connection header received
0808 18:37:57.184301 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:57.184328 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:57.184859 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:57.185271 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:57.185860 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:57.187376 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:57.188275 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:57.188980 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:57.189146 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:57.190048 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:57.190054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:57.190069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:57.190607 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:57.190676 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:57.190707 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:57.190826 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:57.190960 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 18:37:57.191197 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":154,"threads_started":1}
I20260808 18:37:57.192898 18556 log.cc:824] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:57.195123 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:37:57.195268 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34: No bootstrap required, opened a new log
I20260808 18:37:57.199538 18556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260808 18:37:57.199792 18556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:57.199855 18556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e06998401814d1f875954db3b176e34, State: Initialized, Role: FOLLOWER
I20260808 18:37:57.200364 18556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260808 18:37:57.200559 18556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:57.200680 18556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:57.200803 18556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:57.201871 18556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260808 18:37:57.202061 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:57.197197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53235 (local address 127.0.0.1:44713)
0808 18:37:57.197334 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:57.197340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:57.197564 (+ 224us) server_negotiation.cc:407] Connection header received
0808 18:37:57.197634 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:57.197639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:57.197688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:57.197777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:57.198026 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:57.198849 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:57.200059 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:37:57.200384 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:37:57.200498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:37:57.201596 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:57.201600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:57.201602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:57.201617 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:57.201645 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:57.201661 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:57.201749 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:57.201854 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 18:37:57.201915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 18:37:57.202349 18556 leader_election.cc:302] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e06998401814d1f875954db3b176e34; no voters:
I20260808 18:37:57.202775 18556 leader_election.cc:288] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:57.202849 18566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:57.203047 18566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [term 1 LEADER]: Becoming Leader. State: Replica: 7e06998401814d1f875954db3b176e34, State: Running, Role: LEADER
I20260808 18:37:57.203356 18566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } }
I20260808 18:37:57.204142 18556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:57.204653 18566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e06998401814d1f875954db3b176e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260808 18:37:57.204834 18566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:57.204689 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e06998401814d1f875954db3b176e34. Latest consensus state: current_term: 1 leader_uuid: "7e06998401814d1f875954db3b176e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e06998401814d1f875954db3b176e34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44713 } } }
I20260808 18:37:57.205245 18571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:57.205269 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34 [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:57.208618 18571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:57.214358 18571 catalog_manager.cc:1347] Generated new cluster ID: e8f02033c51545d9b260c1bc02641c8a
I20260808 18:37:57.214437 18571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:57.225345 18571 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:57.226289 18571 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:57.234519 18571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e06998401814d1f875954db3b176e34: Generated new TSK 0
I20260808 18:37:57.235339 18571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:57.247960 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18483
2026-08-08T18:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:57 dist-test-slave-fd7l krb5kdc[18587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:57 dist-test-slave-fd7l krb5kdc[18587](info): set up 2 sockets
Aug 08 18:37:57 dist-test-slave-fd7l krb5kdc[18587](info): commencing operation
krb5kdc: starting...
W20260808 18:37:58.352090 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:58 dist-test-slave-fd7l krb5kdc[18587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:58.485821 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:37:58.602617 18601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:37:58.602890 18601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:37:58.602931 18601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:37:58.606971 18601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:37:58.607038 18601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:37:58.607059 18601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:37:58.607075 18601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:37:58.607090 18601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:37:58.612357 18601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:37:58.613514 18601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:37:58.614539 18601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:37:58.623868 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:37:58.623816 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:58.623907 18601 server_base.cc:1034] running on GCE node
W20260808 18:37:58.623845 18612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:37:58.624704 18601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:37:58.625816 18601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:37:58.627019 18601 hybrid_clock.cc:648] HybridClock initialized: now 1786214278626998 us; error 40 us; skew 500 ppm
Aug 08 18:37:58 dist-test-slave-fd7l krb5kdc[18587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:37:58.629843 18601 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:37:58.630703 18601 webserver.cc:460] Webserver started at http://127.0.0.1:41715/ using document root <none> and password file <none>
I20260808 18:37:58.631222 18601 fs_manager.cc:359] Metadata directory not provided
I20260808 18:37:58.631280 18601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:37:58.631443 18601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:37:58.633586 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "bd4d3bded75947b194586b322d0907fd"
format_stamp: "Formatted at 2026-08-08 18:37:58 on dist-test-slave-fd7l"
server_key: "b05a2e95a938df10eb113aa581f3d978"
server_key_iv: "770f877c5a91f8ad5d428f41db165db0"
server_key_version: "encryptionkey@0"
I20260808 18:37:58.634114 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "bd4d3bded75947b194586b322d0907fd"
format_stamp: "Formatted at 2026-08-08 18:37:58 on dist-test-slave-fd7l"
server_key: "b05a2e95a938df10eb113aa581f3d978"
server_key_iv: "770f877c5a91f8ad5d428f41db165db0"
server_key_version: "encryptionkey@0"
I20260808 18:37:58.637799 18601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:37:58.640202 18620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:37:58.641501 18601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:37:58.641657 18601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "bd4d3bded75947b194586b322d0907fd"
format_stamp: "Formatted at 2026-08-08 18:37:58 on dist-test-slave-fd7l"
server_key: "b05a2e95a938df10eb113aa581f3d978"
server_key_iv: "770f877c5a91f8ad5d428f41db165db0"
server_key_version: "encryptionkey@0"
I20260808 18:37:58.641757 18601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:37:58.646482 18601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:37:58.649367 18601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:37:58.649555 18601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:37:58.657238 18601 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38445
I20260808 18:37:58.657354 18675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38445 every 8 connection(s)
I20260808 18:37:58.658255 18601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:37:58.661135 18676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:37:58.663014 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18601
I20260808 18:37:58.663188 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:37:58.663475 28655 external_mini_cluster.cc:1428] Setting key 9a7004bf8312f53ac13b108fabd9f352
I20260808 18:37:58.667877 18676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd: Bootstrap starting.
I20260808 18:37:58.668747 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:58.664860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34859 (local address 127.0.0.1:38445)
0808 18:37:58.665268 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:58.665284 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:58.665319 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:37:58.665962 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:58.665980 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:58.666254 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:58.666525 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:58.667409 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:58.667414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:58.667426 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:58.667437 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:58.667491 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:58.667509 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:58.667628 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:58.667751 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 18:37:58.667919 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260808 18:37:58.670513 18676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd: Neither blocks nor log segments found. Creating new log.
I20260808 18:37:58.671363 18676 log.cc:824] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd: Log is configured to *not* fsync() on all Append() calls
I20260808 18:37:58.671588 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:37:58.673492 18676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd: No bootstrap required, opened a new log
I20260808 18:37:58.677254 18676 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260808 18:37:58.677507 18676 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:37:58.677587 18676 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4d3bded75947b194586b322d0907fd, State: Initialized, Role: FOLLOWER
I20260808 18:37:58.678109 18676 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260808 18:37:58.678233 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:37:58.673873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39693 (local address 127.0.0.1:38445)
0808 18:37:58.673997 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 18:37:58.674001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:37:58.676809 (+ 2808us) server_negotiation.cc:407] Connection header received
0808 18:37:58.676884 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:37:58.676888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:37:58.676925 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:37:58.677010 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:37:58.677788 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:37:58.677792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:37:58.677794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:37:58.677806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:37:58.677835 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:37:58.677849 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:37:58.677940 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 18:37:58.678027 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 18:37:58.678065 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:37:58.678298 18676 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:37:58.678361 18676 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:37:58.678454 18676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:37:58.679351 18676 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260808 18:37:58.679653 18676 leader_election.cc:302] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4d3bded75947b194586b322d0907fd; no voters:
I20260808 18:37:58.679903 18676 leader_election.cc:288] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:37:58.680035 18686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:37:58.680275 18686 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [term 1 LEADER]: Becoming Leader. State: Replica: bd4d3bded75947b194586b322d0907fd, State: Running, Role: LEADER
I20260808 18:37:58.680646 18686 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } }
I20260808 18:37:58.681183 18676 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:37:58.682283 18688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4d3bded75947b194586b322d0907fd. Latest consensus state: current_term: 1 leader_uuid: "bd4d3bded75947b194586b322d0907fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } } }
I20260808 18:37:58.682425 18688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:58.682581 18687 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4d3bded75947b194586b322d0907fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4d3bded75947b194586b322d0907fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38445 } } }
I20260808 18:37:58.682754 18687 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd [sys.catalog]: This master's current role is: LEADER
I20260808 18:37:58.683066 18697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:37:58.686408 18697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:37:58.692102 18697 catalog_manager.cc:1347] Generated new cluster ID: 8d435b73cb364cd28fe63319abaa2f69
I20260808 18:37:58.692207 18697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:37:58.706089 18697 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:37:58.706492 18697 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:37:58.711522 18697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd4d3bded75947b194586b322d0907fd: Generated new TSK 0
I20260808 18:37:58.712215 18697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:37:58.716706 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18601
2026-08-08T18:37:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:37:58 dist-test-slave-fd7l krb5kdc[18708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:37:58 dist-test-slave-fd7l krb5kdc[18708](info): set up 2 sockets
Aug 08 18:37:58 dist-test-slave-fd7l krb5kdc[18708](info): commencing operation
krb5kdc: starting...
W20260808 18:37:59.811474 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:37:59 dist-test-slave-fd7l krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:37:59.947997 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:00.067215 18723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:00.067523 18723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:00.067615 18723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:00.071733 18723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:00.071834 18723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:00.071874 18723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:00.071913 18723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:00.071943 18723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:00.077405 18723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:00.078536 18723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:00.079619 18723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:00.088730 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:00.088828 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:00.088894 18723 server_base.cc:1034] running on GCE node
W20260808 18:38:00.088796 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:00.089648 18723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:00.091009 18723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:00.092190 18723 hybrid_clock.cc:648] HybridClock initialized: now 1786214280092139 us; error 68 us; skew 500 ppm
Aug 08 18:38:00 dist-test-slave-fd7l krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:00.101982 18723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:00.103267 18723 webserver.cc:460] Webserver started at http://127.0.0.1:34901/ using document root <none> and password file <none>
I20260808 18:38:00.103922 18723 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:00.104035 18723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:00.104301 18723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:00.106825 18723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "c1b54fe2a89d4320b29fa58a4417bcc2"
format_stamp: "Formatted at 2026-08-08 18:38:00 on dist-test-slave-fd7l"
server_key: "e75cfbc5ea3dd78b9ac3d0b75c5595bf"
server_key_iv: "95adf70537c9e3b62d5edeec2206efa1"
server_key_version: "encryptionkey@0"
I20260808 18:38:00.107482 18723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "c1b54fe2a89d4320b29fa58a4417bcc2"
format_stamp: "Formatted at 2026-08-08 18:38:00 on dist-test-slave-fd7l"
server_key: "e75cfbc5ea3dd78b9ac3d0b75c5595bf"
server_key_iv: "95adf70537c9e3b62d5edeec2206efa1"
server_key_version: "encryptionkey@0"
I20260808 18:38:00.111780 18723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 18:38:00.114472 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:00.115863 18723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 18:38:00.116034 18723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "c1b54fe2a89d4320b29fa58a4417bcc2"
format_stamp: "Formatted at 2026-08-08 18:38:00 on dist-test-slave-fd7l"
server_key: "e75cfbc5ea3dd78b9ac3d0b75c5595bf"
server_key_iv: "95adf70537c9e3b62d5edeec2206efa1"
server_key_version: "encryptionkey@0"
I20260808 18:38:00.116156 18723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:00.133368 18723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:00.137053 18723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:00.137267 18723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:00.144945 18723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40031
I20260808 18:38:00.144976 18795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40031 every 8 connection(s)
I20260808 18:38:00.145979 18723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:00.149183 18796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:00.154790 18796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2: Bootstrap starting.
I20260808 18:38:00.155834 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18723
I20260808 18:38:00.156106 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:00.156569 28655 external_mini_cluster.cc:1428] Setting key cd76d1efc017fda1b0e9fa9d767fbf95
I20260808 18:38:00.157757 18796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:00.158672 18796 log.cc:824] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:00.160975 18796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2: No bootstrap required, opened a new log
I20260808 18:38:00.164403 18796 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } }
I20260808 18:38:00.164702 18796 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:00.164778 18796 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b54fe2a89d4320b29fa58a4417bcc2, State: Initialized, Role: FOLLOWER
I20260808 18:38:00.165223 18796 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } }
I20260808 18:38:00.165375 18796 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:00.165459 18796 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:00.165560 18796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 18:38:00 dist-test-slave-fd7l krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:38:00.166776 18796 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } }
I20260808 18:38:00.167142 18796 leader_election.cc:302] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1b54fe2a89d4320b29fa58a4417bcc2; no voters:
I20260808 18:38:00.167537 18796 leader_election.cc:288] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:00.167672 18801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:00.167937 18801 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [term 1 LEADER]: Becoming Leader. State: Replica: c1b54fe2a89d4320b29fa58a4417bcc2, State: Running, Role: LEADER
I20260808 18:38:00.168336 18801 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } }
I20260808 18:38:00.169042 18796 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:00.170603 18802 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } } }
I20260808 18:38:00.170615 18803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b54fe2a89d4320b29fa58a4417bcc2. Latest consensus state: current_term: 1 leader_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b54fe2a89d4320b29fa58a4417bcc2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40031 } } }
I20260808 18:38:00.170856 18803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:00.171123 18802 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:00.171633 18814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:00.173491 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:00.158022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:38921 (local address 127.0.0.1:40031)
0808 18:38:00.158527 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:00.158548 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:00.158591 (+ 43us) server_negotiation.cc:407] Connection header received
0808 18:38:00.159401 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:00.159421 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:00.159867 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:00.160236 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:00.161202 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:00.162290 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:00.163240 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:00.163782 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:00.163967 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:00.167366 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:00.167372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:00.167389 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:00.167425 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:00.169646 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:00.170314 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:00.170323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:00.170334 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:00.170502 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:00.170948 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:00.170951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:00.170953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:00.171479 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:00.171715 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:00.172313 (+ 598us) server_negotiation.cc:299] Negotiation successful
0808 18:38:00.172626 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":152,"threads_started":1}
I20260808 18:38:00.174985 18814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:00.176949 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:00.181154 18814 catalog_manager.cc:1347] Generated new cluster ID: 4fa94fa3e5984ce09d9d9384ef1052ee
I20260808 18:38:00.181217 18814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:00.190544 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:00.179012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:34809 (local address 127.0.0.1:40031)
0808 18:38:00.179126 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:00.179130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:00.179353 (+ 223us) server_negotiation.cc:407] Connection header received
0808 18:38:00.179488 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:00.179491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:00.179539 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:00.179611 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:00.180574 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:00.181499 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:00.182886 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:00.183952 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:00.184049 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:00.186191 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:00.186196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:00.186199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:00.186228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:00.188063 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:00.188779 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:00.188783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:00.188785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:00.188882 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:00.189336 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:00.189340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:00.189342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:00.189599 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:00.189733 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:00.190220 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 18:38:00.190346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:38:00.192385 18814 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:00.193825 18814 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:00.203183 18814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1b54fe2a89d4320b29fa58a4417bcc2: Generated new TSK 0
I20260808 18:38:00.204135 18814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:00.208258 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18723
W20260808 18:38:00.217931 18706 connection.cc:535] client connection to 127.0.0.1:40031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40031): Connection reset by peer (error 104)
2026-08-08T18:38:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:00 dist-test-slave-fd7l krb5kdc[18827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:00 dist-test-slave-fd7l krb5kdc[18827](info): set up 2 sockets
Aug 08 18:38:00 dist-test-slave-fd7l krb5kdc[18827](info): commencing operation
krb5kdc: starting...
W20260808 18:38:01.367090 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:01 dist-test-slave-fd7l krb5kdc[18827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:38:01.507655 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:01.621893 18840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:01.622225 18840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:01.622309 18840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:01.626258 18840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:01.626346 18840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:01.626391 18840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:01.626430 18840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:01.626461 18840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:01.631736 18840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:01.632982 18840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:01.634090 18840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:01.643121 18845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:01.643291 18840 server_base.cc:1034] running on GCE node
W20260808 18:38:01.643208 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:01.643121 18846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:01.643932 18840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:01.645044 18840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:01.646227 18840 hybrid_clock.cc:648] HybridClock initialized: now 1786214281646213 us; error 27 us; skew 500 ppm
Aug 08 18:38:01 dist-test-slave-fd7l krb5kdc[18827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:01.648978 18840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:01.649978 18840 webserver.cc:460] Webserver started at http://127.0.0.1:39203/ using document root <none> and password file <none>
I20260808 18:38:01.650492 18840 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:01.650568 18840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:01.650738 18840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:01.652782 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909"
format_stamp: "Formatted at 2026-08-08 18:38:01 on dist-test-slave-fd7l"
server_key: "89224c225aa14f3e0431b89f595e6847"
server_key_iv: "5a81b80580b0f52c1ce3d49297512f76"
server_key_version: "encryptionkey@0"
I20260808 18:38:01.653298 18840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909"
format_stamp: "Formatted at 2026-08-08 18:38:01 on dist-test-slave-fd7l"
server_key: "89224c225aa14f3e0431b89f595e6847"
server_key_iv: "5a81b80580b0f52c1ce3d49297512f76"
server_key_version: "encryptionkey@0"
I20260808 18:38:01.656911 18840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:38:01.659395 18855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:01.660825 18840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:38:01.660948 18840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909"
format_stamp: "Formatted at 2026-08-08 18:38:01 on dist-test-slave-fd7l"
server_key: "89224c225aa14f3e0431b89f595e6847"
server_key_iv: "5a81b80580b0f52c1ce3d49297512f76"
server_key_version: "encryptionkey@0"
I20260808 18:38:01.661059 18840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:01.681051 18840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:01.684214 18840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:01.684429 18840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:01.693686 18840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39519
I20260808 18:38:01.694041 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39519 every 8 connection(s)
I20260808 18:38:01.694898 18840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:01.697779 18912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:01.703648 18912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: Bootstrap starting.
I20260808 18:38:01.705184 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18840
I20260808 18:38:01.705370 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:01.705708 28655 external_mini_cluster.cc:1428] Setting key a3086608708b65142e1b92b57374426d
I20260808 18:38:01.706449 18912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:01.707291 18912 log.cc:824] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:01.709762 18912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: No bootstrap required, opened a new log
I20260808 18:38:01.714221 18912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } }
I20260808 18:38:01.714474 18912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:01.714562 18912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1179ff1b5b5b4f8d8c2ff7a2aa833909, State: Initialized, Role: FOLLOWER
I20260808 18:38:01.715082 18912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } }
I20260808 18:38:01.715215 18912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:01.715301 18912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:01.715410 18912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:01.715965 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:01.707100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:42225 (local address 127.0.0.1:39519)
0808 18:38:01.707604 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:01.707622 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:01.707662 (+ 40us) server_negotiation.cc:407] Connection header received
0808 18:38:01.708642 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:01.708666 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:01.709049 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:01.709363 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:01.709764 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:01.710886 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:01.712201 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:01.712819 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:01.712959 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:01.713949 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:01.713954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:01.713968 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:01.714372 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:01.714429 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:01.714451 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:01.714683 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:01.714873 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 18:38:01.715026 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":152,"threads_started":1}
I20260808 18:38:01.716414 18912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } }
I20260808 18:38:01.716775 18912 leader_election.cc:302] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1179ff1b5b5b4f8d8c2ff7a2aa833909; no voters:
I20260808 18:38:01.717124 18912 leader_election.cc:288] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:01.717175 18919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:01.717460 18919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [term 1 LEADER]: Becoming Leader. State: Replica: 1179ff1b5b5b4f8d8c2ff7a2aa833909, State: Running, Role: LEADER
I20260808 18:38:01.717811 18919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } }
I20260808 18:38:01.718485 18912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:01.719216 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:01.721611 18920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } } }
I20260808 18:38:01.721752 18920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:01.721947 18921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1179ff1b5b5b4f8d8c2ff7a2aa833909. Latest consensus state: current_term: 1 leader_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1179ff1b5b5b4f8d8c2ff7a2aa833909" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39519 } } }
I20260808 18:38:01.722064 18921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909 [sys.catalog]: This master's current role is: LEADER
W20260808 18:38:01.724048 18937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:01.724135 18937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:01.724236 18941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:01.725085 18941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:01.726874 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:01.721672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:55389 (local address 127.0.0.1:39519)
0808 18:38:01.721778 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:01.721783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:01.722434 (+ 651us) server_negotiation.cc:407] Connection header received
0808 18:38:01.722599 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:01.722603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:01.722650 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:01.722725 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:01.723028 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:01.723776 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:01.724939 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:01.725273 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:01.725382 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:01.726363 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:01.726367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:01.726370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:01.726383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:01.726414 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:01.726427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:01.726520 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:01.726713 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 18:38:01.726750 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 18:38:01.731369 18941 catalog_manager.cc:1347] Generated new cluster ID: 4bd3c3a3595a464da26963658b5fc89d
I20260808 18:38:01.731436 18941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:01.745644 18941 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:01.746572 18941 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:01.759804 18941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1179ff1b5b5b4f8d8c2ff7a2aa833909: Generated new TSK 0
I20260808 18:38:01.760591 18941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:01.770537 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18840
W20260808 18:38:01.779376 18825 connection.cc:535] client connection to 127.0.0.1:39519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39519): Connection reset by peer (error 104)
2026-08-08T18:38:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1561 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:01 dist-test-slave-fd7l krb5kdc[18946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:01 dist-test-slave-fd7l krb5kdc[18946](info): set up 2 sockets
Aug 08 18:38:01 dist-test-slave-fd7l krb5kdc[18946](info): commencing operation
krb5kdc: starting...
W20260808 18:38:02.859355 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:02 dist-test-slave-fd7l krb5kdc[18946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:38:02.993415 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:03.109647 18961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:03.109957 18961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:03.110033 18961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:03.114105 18961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:03.114231 18961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:03.114269 18961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:03.114323 18961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:03.114362 18961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:03.119737 18961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:03.120970 18961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:03.122026 18961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:03.131000 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:03.131124 18967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:03.131052 18968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:03.131244 18961 server_base.cc:1034] running on GCE node
I20260808 18:38:03.131938 18961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:03.133186 18961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:03.134369 18961 hybrid_clock.cc:648] HybridClock initialized: now 1786214283134344 us; error 42 us; skew 500 ppm
Aug 08 18:38:03 dist-test-slave-fd7l krb5kdc[18946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:03.137197 18961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:03.138119 18961 webserver.cc:460] Webserver started at http://127.0.0.1:40353/ using document root <none> and password file <none>
I20260808 18:38:03.138696 18961 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:03.138783 18961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:03.139017 18961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:03.141211 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "23f6a6195dee4a4c8289f3929c98f170"
format_stamp: "Formatted at 2026-08-08 18:38:03 on dist-test-slave-fd7l"
server_key: "dc31e140c0babeb006dc66e0238ba540"
server_key_iv: "4927bb502070f7e38df1ecbdc5178d4a"
server_key_version: "encryptionkey@0"
I20260808 18:38:03.141772 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "23f6a6195dee4a4c8289f3929c98f170"
format_stamp: "Formatted at 2026-08-08 18:38:03 on dist-test-slave-fd7l"
server_key: "dc31e140c0babeb006dc66e0238ba540"
server_key_iv: "4927bb502070f7e38df1ecbdc5178d4a"
server_key_version: "encryptionkey@0"
I20260808 18:38:03.145519 18961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:38:03.148170 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:03.149505 18961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 18:38:03.149691 18961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "23f6a6195dee4a4c8289f3929c98f170"
format_stamp: "Formatted at 2026-08-08 18:38:03 on dist-test-slave-fd7l"
server_key: "dc31e140c0babeb006dc66e0238ba540"
server_key_iv: "4927bb502070f7e38df1ecbdc5178d4a"
server_key_version: "encryptionkey@0"
I20260808 18:38:03.149806 18961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:03.154577 18961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:03.157861 18961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:03.158061 18961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:03.165541 18961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40427
I20260808 18:38:03.165572 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40427 every 8 connection(s)
I20260808 18:38:03.166512 18961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:03.169409 19031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:03.171214 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 18961
I20260808 18:38:03.171356 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:03.171641 28655 external_mini_cluster.cc:1428] Setting key f61bcb6aea90949a2cf64cca09a18f6a
I20260808 18:38:03.176147 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170: Bootstrap starting.
I20260808 18:38:03.177156 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:03.173113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:42335 (local address 127.0.0.1:40427)
0808 18:38:03.173580 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:03.173596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:03.173641 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:38:03.174328 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:03.174347 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:03.174637 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:03.174971 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:03.175819 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:03.175827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:03.175842 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:03.175857 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:03.175927 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:03.175948 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:03.176106 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:03.176185 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 18:38:03.176360 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":155,"threads_started":1}
I20260808 18:38:03.178702 19031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:03.179621 19031 log.cc:824] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:03.180348 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:03.181722 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170: No bootstrap required, opened a new log
I20260808 18:38:03.184193 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:03.182524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:43865 (local address 127.0.0.1:40427)
0808 18:38:03.182645 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:03.182649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:03.182809 (+ 160us) server_negotiation.cc:407] Connection header received
0808 18:38:03.182912 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:03.182915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:03.182949 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:03.183023 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:03.183815 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:03.183819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:03.183822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:03.183833 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:03.183861 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:03.183873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:03.183957 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:03.184027 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 18:38:03.184072 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:38:03.186031 19031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260808 18:38:03.186259 19031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:03.186295 19031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f6a6195dee4a4c8289f3929c98f170, State: Initialized, Role: FOLLOWER
I20260808 18:38:03.186707 19031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260808 18:38:03.186830 19031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:03.186877 19031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:03.186949 19031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:03.187834 19031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260808 18:38:03.188141 19031 leader_election.cc:302] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f6a6195dee4a4c8289f3929c98f170; no voters:
I20260808 18:38:03.188421 19031 leader_election.cc:288] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:03.188659 19042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:03.188915 19042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [term 1 LEADER]: Becoming Leader. State: Replica: 23f6a6195dee4a4c8289f3929c98f170, State: Running, Role: LEADER
I20260808 18:38:03.189441 19042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260808 18:38:03.189931 19031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:03.191290 19044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f6a6195dee4a4c8289f3929c98f170. Latest consensus state: current_term: 1 leader_uuid: "23f6a6195dee4a4c8289f3929c98f170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260808 18:38:03.191426 19044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:03.191771 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f6a6195dee4a4c8289f3929c98f170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f6a6195dee4a4c8289f3929c98f170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260808 18:38:03.191854 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:03.192222 19054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:03.195353 19054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:03.201690 19054 catalog_manager.cc:1347] Generated new cluster ID: 64b62f1700b745b18e6dc9bc95fb67d5
I20260808 18:38:03.201784 19054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:03.235904 19054 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:03.236289 19054 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:03.241557 19054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f6a6195dee4a4c8289f3929c98f170: Generated new TSK 0
I20260808 18:38:03.242810 19054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:03.255367 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 18961
2026-08-08T18:38:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:03 dist-test-slave-fd7l krb5kdc[19066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:03 dist-test-slave-fd7l krb5kdc[19066](info): set up 2 sockets
Aug 08 18:38:03 dist-test-slave-fd7l krb5kdc[19066](info): commencing operation
krb5kdc: starting...
W20260808 18:38:04.350692 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:38:04.481709 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:04.599918 19081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:04.600193 19081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:04.600235 19081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:04.604287 19081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:04.604350 19081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:04.604373 19081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:04.604393 19081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:04.604416 19081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:04.609611 19081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:04.610625 19081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:04.611595 19081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:04.620638 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:04.620647 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:04.620721 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:04.621025 19081 server_base.cc:1034] running on GCE node
I20260808 18:38:04.621690 19081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:04.622816 19081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:04.623996 19081 hybrid_clock.cc:648] HybridClock initialized: now 1786214284623966 us; error 43 us; skew 500 ppm
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:04.626991 19081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:04.627936 19081 webserver.cc:460] Webserver started at http://127.0.0.1:44907/ using document root <none> and password file <none>
I20260808 18:38:04.628563 19081 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:04.628633 19081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:04.628803 19081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:04.630795 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "5d793efe1e3c4fa9bab04711cb4956b5"
format_stamp: "Formatted at 2026-08-08 18:38:04 on dist-test-slave-fd7l"
server_key: "d9987bc748075c8f7935350e36f4a26a"
server_key_iv: "206ad1ab9ebf9d3a63687478ce85f200"
server_key_version: "encryptionkey@0"
I20260808 18:38:04.631326 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "5d793efe1e3c4fa9bab04711cb4956b5"
format_stamp: "Formatted at 2026-08-08 18:38:04 on dist-test-slave-fd7l"
server_key: "d9987bc748075c8f7935350e36f4a26a"
server_key_iv: "206ad1ab9ebf9d3a63687478ce85f200"
server_key_version: "encryptionkey@0"
I20260808 18:38:04.635059 19081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:38:04.637449 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:04.638885 19081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:38:04.639043 19081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "5d793efe1e3c4fa9bab04711cb4956b5"
format_stamp: "Formatted at 2026-08-08 18:38:04 on dist-test-slave-fd7l"
server_key: "d9987bc748075c8f7935350e36f4a26a"
server_key_iv: "206ad1ab9ebf9d3a63687478ce85f200"
server_key_version: "encryptionkey@0"
I20260808 18:38:04.639169 19081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:04.653187 19081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:04.656107 19081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:04.656320 19081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:04.664207 19081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35013
I20260808 18:38:04.664233 19152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35013 every 8 connection(s)
I20260808 18:38:04.665241 19081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:04.668041 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:04.668915 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19081
I20260808 18:38:04.669076 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:04.669389 28655 external_mini_cluster.cc:1428] Setting key f3b251ed622d76a5531f1f241cde8840
I20260808 18:38:04.674125 19153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: Bootstrap starting.
I20260808 18:38:04.676368 19153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:04.677266 19153 log.cc:824] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 18:38:04.679467 19153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: No bootstrap required, opened a new log
I20260808 18:38:04.683400 19153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } }
I20260808 18:38:04.683656 19153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:04.683759 19153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d793efe1e3c4fa9bab04711cb4956b5, State: Initialized, Role: FOLLOWER
I20260808 18:38:04.684278 19153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } }
I20260808 18:38:04.684458 19153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:04.684573 19153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:04.684682 19153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:04.685385 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:04.670829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:39409 (local address 127.0.0.1:35013)
0808 18:38:04.671297 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:04.671313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:04.671352 (+ 39us) server_negotiation.cc:407] Connection header received
0808 18:38:04.672066 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:04.672084 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:04.672454 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:04.672899 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:04.673808 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:04.674928 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:04.675867 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:04.676383 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:04.676607 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:04.679891 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:04.679903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:04.679917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:04.679947 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:04.681908 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:04.682556 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:04.682562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:04.682567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:04.682690 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:04.683064 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:04.683072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:04.683073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:04.683489 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:04.683743 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:04.684153 (+ 410us) server_negotiation.cc:299] Negotiation successful
0808 18:38:04.684391 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":148,"threads_started":1}
I20260808 18:38:04.685669 19153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } }
I20260808 18:38:04.686028 19153 leader_election.cc:302] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d793efe1e3c4fa9bab04711cb4956b5; no voters:
I20260808 18:38:04.686353 19153 leader_election.cc:288] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:04.686548 19159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:04.686856 19159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [term 1 LEADER]: Becoming Leader. State: Replica: 5d793efe1e3c4fa9bab04711cb4956b5, State: Running, Role: LEADER
I20260808 18:38:04.687193 19159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } }
I20260808 18:38:04.687664 19153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:04.688824 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:04.689009 19160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } } }
I20260808 18:38:04.689148 19160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:04.689410 19160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d793efe1e3c4fa9bab04711cb4956b5. Latest consensus state: current_term: 1 leader_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d793efe1e3c4fa9bab04711cb4956b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35013 } } }
I20260808 18:38:04.689589 19160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5 [sys.catalog]: This master's current role is: LEADER
W20260808 18:38:04.693506 19174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:04.693588 19174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:04.693684 19170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:04.694618 19170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:04.700807 19170 catalog_manager.cc:1347] Generated new cluster ID: b2f8016158d141a59e0e47208038b304
I20260808 18:38:04.700872 19170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:04.702065 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:04.691146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:51919 (local address 127.0.0.1:35013)
0808 18:38:04.691283 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:04.691288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:04.691453 (+ 165us) server_negotiation.cc:407] Connection header received
0808 18:38:04.691616 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:04.691620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:04.691666 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:04.691761 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:04.692475 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:04.693088 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:04.694118 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:04.694390 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:04.694508 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:04.696105 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:04.696109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:04.696112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:04.696139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:04.699670 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:04.700209 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:04.700213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:04.700214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:04.700307 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:04.700857 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:04.700861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:04.700862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:04.701103 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:04.701232 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:04.701781 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 18:38:04.701892 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 18:38:04.749819 19170 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:04.750814 19170 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:04.758172 19170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d793efe1e3c4fa9bab04711cb4956b5: Generated new TSK 0
I20260808 18:38:04.758944 19170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:04.763824 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19081
W20260808 18:38:04.773357 19064 connection.cc:535] client connection to 127.0.0.1:35013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35013): Connection reset by peer (error 104)
2026-08-08T18:38:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19186](info): set up 2 sockets
Aug 08 18:38:04 dist-test-slave-fd7l krb5kdc[19186](info): commencing operation
krb5kdc: starting...
W20260808 18:38:05.848877 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:05 dist-test-slave-fd7l krb5kdc[19186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:38:05.981091 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:06.102844 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:06.103123 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:06.103168 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:06.107347 19199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:06.107414 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:06.107434 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:06.107450 19199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:06.107465 19199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:06.112775 19199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:06.113817 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:06.114861 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:06.123695 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:06.123690 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:06.123690 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:06.124096 19199 server_base.cc:1034] running on GCE node
I20260808 18:38:06.124725 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:06.125888 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:06.127105 19199 hybrid_clock.cc:648] HybridClock initialized: now 1786214286127072 us; error 44 us; skew 500 ppm
Aug 08 18:38:06 dist-test-slave-fd7l krb5kdc[19186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:06.134984 19199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:06.135887 19199 webserver.cc:460] Webserver started at http://127.0.0.1:38819/ using document root <none> and password file <none>
I20260808 18:38:06.136421 19199 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:06.136497 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:06.136752 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:06.138757 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "8e27f6a12f6a41c2b2148a23fedfb047"
format_stamp: "Formatted at 2026-08-08 18:38:06 on dist-test-slave-fd7l"
server_key: "0e83b5b0d6f4a907a42658db57a23fe2"
server_key_iv: "d94cbc375f9c2d461d16da98bff64151"
server_key_version: "encryptionkey@0"
I20260808 18:38:06.139262 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "8e27f6a12f6a41c2b2148a23fedfb047"
format_stamp: "Formatted at 2026-08-08 18:38:06 on dist-test-slave-fd7l"
server_key: "0e83b5b0d6f4a907a42658db57a23fe2"
server_key_iv: "d94cbc375f9c2d461d16da98bff64151"
server_key_version: "encryptionkey@0"
I20260808 18:38:06.142993 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 18:38:06.145319 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:06.146648 19199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:38:06.146802 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "8e27f6a12f6a41c2b2148a23fedfb047"
format_stamp: "Formatted at 2026-08-08 18:38:06 on dist-test-slave-fd7l"
server_key: "0e83b5b0d6f4a907a42658db57a23fe2"
server_key_iv: "d94cbc375f9c2d461d16da98bff64151"
server_key_version: "encryptionkey@0"
I20260808 18:38:06.146920 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:06.165973 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:06.169301 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:06.169510 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:06.176812 19199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39771
I20260808 18:38:06.176841 19273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39771 every 8 connection(s)
I20260808 18:38:06.177779 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:06.178699 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19199
I20260808 18:38:06.178843 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:06.179167 28655 external_mini_cluster.cc:1428] Setting key 24a99f9afcde832d8e0c72f17d8815c8
I20260808 18:38:06.181226 19274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:06.186936 19274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047: Bootstrap starting.
I20260808 18:38:06.188531 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:06.180741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:57161 (local address 127.0.0.1:39771)
0808 18:38:06.181186 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:06.181201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:06.181239 (+ 38us) server_negotiation.cc:407] Connection header received
0808 18:38:06.181986 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:06.182003 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:06.182365 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:06.182709 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:06.183155 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:06.184194 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:06.185164 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:06.185642 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:06.185814 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:06.186745 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:06.186749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:06.186762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:06.187196 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:06.187252 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:06.187277 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:06.187418 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:06.187623 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 18:38:06.187762 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
I20260808 18:38:06.189392 19274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:06.190140 19274 log.cc:824] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:06.191439 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:06.192444 19274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047: No bootstrap required, opened a new log
I20260808 18:38:06.196342 19274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } }
I20260808 18:38:06.196676 19274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:06.196763 19274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e27f6a12f6a41c2b2148a23fedfb047, State: Initialized, Role: FOLLOWER
I20260808 18:38:06.197250 19274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } }
I20260808 18:38:06.197377 19274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:06.197511 19274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:06.197639 19274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:06.198446 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:06.193756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:49253 (local address 127.0.0.1:39771)
0808 18:38:06.193884 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:06.193890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:06.194070 (+ 180us) server_negotiation.cc:407] Connection header received
0808 18:38:06.194179 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:06.194181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:06.194219 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:06.194285 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:06.194591 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:06.195472 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:06.196626 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:06.196971 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:06.197098 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:06.197970 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:06.197974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:06.197977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:06.197990 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:06.198022 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:06.198033 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:06.198145 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:06.198268 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 18:38:06.198306 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:38:06.198796 19274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } }
I20260808 18:38:06.199168 19274 leader_election.cc:302] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e27f6a12f6a41c2b2148a23fedfb047; no voters:
I20260808 18:38:06.199509 19274 leader_election.cc:288] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:06.199610 19284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:06.199878 19284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [term 1 LEADER]: Becoming Leader. State: Replica: 8e27f6a12f6a41c2b2148a23fedfb047, State: Running, Role: LEADER
I20260808 18:38:06.200258 19284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } }
I20260808 18:38:06.200954 19274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:06.202006 19285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } } }
I20260808 18:38:06.202162 19285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:06.202519 19286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e27f6a12f6a41c2b2148a23fedfb047. Latest consensus state: current_term: 1 leader_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e27f6a12f6a41c2b2148a23fedfb047" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39771 } } }
I20260808 18:38:06.202575 19294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:06.202652 19286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:06.205439 19294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:06.211277 19294 catalog_manager.cc:1347] Generated new cluster ID: e1a042ffc9bf427781f2bcad9d51c8ff
I20260808 18:38:06.211350 19294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:06.235927 19294 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:06.237095 19294 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:06.242027 19294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e27f6a12f6a41c2b2148a23fedfb047: Generated new TSK 0
I20260808 18:38:06.242781 19294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:06.253108 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19199
W20260808 18:38:06.262125 19184 connection.cc:535] client connection to 127.0.0.1:39771 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39771): Connection reset by peer (error 104)
2026-08-08T18:38:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:06 dist-test-slave-fd7l krb5kdc[19307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:06 dist-test-slave-fd7l krb5kdc[19307](info): set up 2 sockets
Aug 08 18:38:06 dist-test-slave-fd7l krb5kdc[19307](info): commencing operation
krb5kdc: starting...
W20260808 18:38:07.348405 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:07 dist-test-slave-fd7l krb5kdc[19307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 18:38:07.482096 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:07.600669 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:07.600999 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:07.601082 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:07.605119 19320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:07.605218 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:07.605262 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:07.605326 19320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:07.605371 19320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:07.610739 19320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:07.611907 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:07.613063 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:07.621913 19328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:07.621889 19326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:07.622234 19320 server_base.cc:1034] running on GCE node
W20260808 18:38:07.621889 19325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:07.622864 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:07.624053 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:07.625248 19320 hybrid_clock.cc:648] HybridClock initialized: now 1786214287625213 us; error 50 us; skew 500 ppm
Aug 08 18:38:07 dist-test-slave-fd7l krb5kdc[19307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:07.628199 19320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 18:38:07.629185 19320 webserver.cc:460] Webserver started at http://127.0.0.1:36805/ using document root <none> and password file <none>
I20260808 18:38:07.629762 19320 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:07.629853 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:07.630100 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:07.632206 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "3bed779c029e476ca5941b33a713d960"
format_stamp: "Formatted at 2026-08-08 18:38:07 on dist-test-slave-fd7l"
server_key: "167c788efae357bbe1734eb867906bca"
server_key_iv: "a3dbfa4e9fa75c1109feac2db9f4235b"
server_key_version: "encryptionkey@0"
I20260808 18:38:07.632790 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "3bed779c029e476ca5941b33a713d960"
format_stamp: "Formatted at 2026-08-08 18:38:07 on dist-test-slave-fd7l"
server_key: "167c788efae357bbe1734eb867906bca"
server_key_iv: "a3dbfa4e9fa75c1109feac2db9f4235b"
server_key_version: "encryptionkey@0"
I20260808 18:38:07.636660 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:38:07.639074 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:07.640324 19320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:38:07.640470 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "3bed779c029e476ca5941b33a713d960"
format_stamp: "Formatted at 2026-08-08 18:38:07 on dist-test-slave-fd7l"
server_key: "167c788efae357bbe1734eb867906bca"
server_key_iv: "a3dbfa4e9fa75c1109feac2db9f4235b"
server_key_version: "encryptionkey@0"
I20260808 18:38:07.640628 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:07.645272 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:07.648231 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:07.648415 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:07.655591 19320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36593
I20260808 18:38:07.655619 19392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36593 every 8 connection(s)
I20260808 18:38:07.656597 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:07.659479 19393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:07.660099 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19320
I20260808 18:38:07.660249 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:07.660579 28655 external_mini_cluster.cc:1428] Setting key 3c5652a4d0c97d91cb5964924dba41e0
I20260808 18:38:07.666540 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:07.662076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:38665 (local address 127.0.0.1:36593)
0808 18:38:07.662546 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:07.662566 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:07.662614 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:38:07.663483 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:07.663508 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:07.663869 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:07.664250 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:07.665089 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:07.665095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:07.665110 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:07.665123 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:07.665191 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:07.665217 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:07.665376 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:07.665446 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 18:38:07.665611 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260808 18:38:07.666577 19393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: Bootstrap starting.
I20260808 18:38:07.668922 19393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:07.669498 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:07.669858 19393 log.cc:824] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:07.672204 19393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: No bootstrap required, opened a new log
I20260808 18:38:07.673548 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:07.671777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:46677 (local address 127.0.0.1:36593)
0808 18:38:07.671909 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:07.671913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:07.672103 (+ 190us) server_negotiation.cc:407] Connection header received
0808 18:38:07.672240 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:07.672242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:07.672271 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:07.672373 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:07.673149 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:07.673151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:07.673154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:07.673165 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:07.673190 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:07.673200 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:07.673272 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:07.673378 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 18:38:07.673411 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:38:07.676318 19393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } }
I20260808 18:38:07.676614 19393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:07.676692 19393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bed779c029e476ca5941b33a713d960, State: Initialized, Role: FOLLOWER
I20260808 18:38:07.677168 19393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } }
I20260808 18:38:07.677328 19393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:07.677403 19393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:07.677506 19393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:07.678485 19393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } }
I20260808 18:38:07.678859 19393 leader_election.cc:302] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bed779c029e476ca5941b33a713d960; no voters:
I20260808 18:38:07.679183 19393 leader_election.cc:288] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:07.679466 19404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:07.679800 19404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [term 1 LEADER]: Becoming Leader. State: Replica: 3bed779c029e476ca5941b33a713d960, State: Running, Role: LEADER
I20260808 18:38:07.680152 19404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } }
I20260808 18:38:07.680558 19393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:07.682001 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bed779c029e476ca5941b33a713d960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } } }
I20260808 18:38:07.682159 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:07.682559 19406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bed779c029e476ca5941b33a713d960. Latest consensus state: current_term: 1 leader_uuid: "3bed779c029e476ca5941b33a713d960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bed779c029e476ca5941b33a713d960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36593 } } }
I20260808 18:38:07.682652 19406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960 [sys.catalog]: This master's current role is: LEADER
W20260808 18:38:07.685956 19419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:07.686033 19419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:07.686146 19415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:07.686856 19415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:07.692813 19415 catalog_manager.cc:1347] Generated new cluster ID: e4008be6b1394529860baf5b1f86a98d
I20260808 18:38:07.692889 19415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:07.706612 19415 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:07.706916 19415 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:07.712098 19415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bed779c029e476ca5941b33a713d960: Generated new TSK 0
I20260808 18:38:07.712827 19415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:07.717398 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19320
2026-08-08T18:38:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:07 dist-test-slave-fd7l krb5kdc[19426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:07 dist-test-slave-fd7l krb5kdc[19426](info): set up 2 sockets
Aug 08 18:38:07 dist-test-slave-fd7l krb5kdc[19426](info): commencing operation
krb5kdc: starting...
W20260808 18:38:08.871421 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:08 dist-test-slave-fd7l krb5kdc[19426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:09.003058 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43843
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:09.118997 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:09.119259 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:09.119328 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:09.123377 19440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:09.123438 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:09.123457 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:09.123523 19440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:09.123561 19440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:09.128911 19440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:43843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:09.129959 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:09.130987 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:09.139847 19448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:09.139886 19446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:09.139886 19445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:09.140070 19440 server_base.cc:1034] running on GCE node
I20260808 18:38:09.140787 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:09.141898 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:09.143076 19440 hybrid_clock.cc:648] HybridClock initialized: now 1786214289143051 us; error 36 us; skew 500 ppm
Aug 08 18:38:09 dist-test-slave-fd7l krb5kdc[19426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:09.146085 19440 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:09.146934 19440 webserver.cc:460] Webserver started at http://127.27.251.254:44985/ using document root <none> and password file <none>
I20260808 18:38:09.147509 19440 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:09.147571 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:09.147742 19440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:09.149854 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "45ad9555ef964bd7bbb82ade345cc471"
format_stamp: "Formatted at 2026-08-08 18:38:09 on dist-test-slave-fd7l"
server_key: "7deb79ad2980365a64bda5a940039dbf"
server_key_iv: "c015c2e6f367f62945f77f27eb9b3ca3"
server_key_version: "encryptionkey@0"
I20260808 18:38:09.150383 19440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "45ad9555ef964bd7bbb82ade345cc471"
format_stamp: "Formatted at 2026-08-08 18:38:09 on dist-test-slave-fd7l"
server_key: "7deb79ad2980365a64bda5a940039dbf"
server_key_iv: "c015c2e6f367f62945f77f27eb9b3ca3"
server_key_version: "encryptionkey@0"
I20260808 18:38:09.154168 19440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 18:38:09.156636 19455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:09.157845 19440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 18:38:09.157974 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "45ad9555ef964bd7bbb82ade345cc471"
format_stamp: "Formatted at 2026-08-08 18:38:09 on dist-test-slave-fd7l"
server_key: "7deb79ad2980365a64bda5a940039dbf"
server_key_iv: "c015c2e6f367f62945f77f27eb9b3ca3"
server_key_version: "encryptionkey@0"
I20260808 18:38:09.158094 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:09.182550 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:09.185675 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:09.185889 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:09.193266 19440 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:43843
I20260808 18:38:09.193292 19508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:43843 every 8 connection(s)
I20260808 18:38:09.194227 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:09.197050 19509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:09.200063 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19440
I20260808 18:38:09.200232 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:09.200553 28655 external_mini_cluster.cc:1428] Setting key 57c1538703aa1c704e978f836a29b795
I20260808 18:38:09.203380 19509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471: Bootstrap starting.
I20260808 18:38:09.205675 19509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:09.206426 19509 log.cc:824] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:09.208444 19509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471: No bootstrap required, opened a new log
Aug 08 18:38:09 dist-test-slave-fd7l krb5kdc[19426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:09.211637 19509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } }
I20260808 18:38:09.211951 19509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:09.212033 19509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ad9555ef964bd7bbb82ade345cc471, State: Initialized, Role: FOLLOWER
I20260808 18:38:09.212566 19509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } }
I20260808 18:38:09.212713 19509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:09.212795 19509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:09.212893 19509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:09.213779 19509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } }
I20260808 18:38:09.214118 19509 leader_election.cc:302] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ad9555ef964bd7bbb82ade345cc471; no voters:
I20260808 18:38:09.214463 19509 leader_election.cc:288] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:09.214617 19514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:09.214818 19514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [term 1 LEADER]: Becoming Leader. State: Replica: 45ad9555ef964bd7bbb82ade345cc471, State: Running, Role: LEADER
I20260808 18:38:09.215179 19514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } }
I20260808 18:38:09.215835 19509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:09.216645 19514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ad9555ef964bd7bbb82ade345cc471. Latest consensus state: current_term: 1 leader_uuid: "45ad9555ef964bd7bbb82ade345cc471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } } }
I20260808 18:38:09.216811 19514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:09.217376 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:09.202032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33153 (local address 127.27.251.254:43843)
0808 18:38:09.202596 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:09.202612 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:09.202650 (+ 38us) server_negotiation.cc:407] Connection header received
0808 18:38:09.203281 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:09.203297 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:09.203704 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:09.204069 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:09.204978 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:09.206051 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:09.206976 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:09.207447 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:09.207616 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:09.210302 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:09.210307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:09.210321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:09.210348 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:09.213149 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:09.213711 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:09.213718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:09.213724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:09.213876 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:09.214243 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:09.214246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:09.214248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:09.214819 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:09.215755 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:09.215942 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 18:38:09.216196 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":195,"threads_started":1}
I20260808 18:38:09.217505 19516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ad9555ef964bd7bbb82ade345cc471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ad9555ef964bd7bbb82ade345cc471" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 43843 } } }
I20260808 18:38:09.217614 19516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:09.217926 19523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:09.220947 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:09.221797 19523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:09.228914 19523 catalog_manager.cc:1347] Generated new cluster ID: 6268a9d503714d18a82f688d5480f7aa
I20260808 18:38:09.228994 19523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:09.231498 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:09.223242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47921 (local address 127.27.251.254:43843)
0808 18:38:09.223350 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:09.223354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:09.223649 (+ 295us) server_negotiation.cc:407] Connection header received
0808 18:38:09.223779 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:09.223784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:09.223835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:09.223941 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:09.224657 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:09.225186 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:09.226014 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:09.226296 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:09.226409 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:09.227856 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:09.227859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:09.227866 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:09.227890 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:09.229581 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:09.230152 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:09.230155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:09.230157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:09.230235 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:09.230567 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:09.230570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:09.230572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:09.230764 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:09.230868 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:09.231253 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 18:38:09.231347 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 18:38:09.243901 19523 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:09.244885 19523 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:09.255470 19523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ad9555ef964bd7bbb82ade345cc471: Generated new TSK 0
I20260808 18:38:09.256388 19523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:09.261492 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19440
W20260808 18:38:09.269984 19424 connection.cc:535] client connection to 127.27.251.254:43843 recv error: Network error: failed to read from TLS socket (remote: 127.27.251.254:43843): Connection reset by peer (error 104)
2026-08-08T18:38:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:09 dist-test-slave-fd7l krb5kdc[19542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:09 dist-test-slave-fd7l krb5kdc[19542](info): set up 2 sockets
Aug 08 18:38:09 dist-test-slave-fd7l krb5kdc[19542](info): commencing operation
krb5kdc: starting...
W20260808 18:38:10.375325 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:10 dist-test-slave-fd7l krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:10.507365 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35989
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:10.621675 19555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:10.622020 19555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:10.622108 19555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:10.626206 19555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:10.626317 19555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:10.626356 19555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:10.626411 19555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:10.626449 19555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:10.631934 19555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:35989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:10.633208 19555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:10.634307 19555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:10.643328 19560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:10.643395 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:10.643309 19561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:10.643651 19555 server_base.cc:1034] running on GCE node
I20260808 18:38:10.644318 19555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:10.645556 19555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:10.646756 19555 hybrid_clock.cc:648] HybridClock initialized: now 1786214290646726 us; error 50 us; skew 500 ppm
Aug 08 18:38:10 dist-test-slave-fd7l krb5kdc[19542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214290, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:10.649672 19555 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:10.650589 19555 webserver.cc:460] Webserver started at http://127.27.251.254:35365/ using document root <none> and password file <none>
I20260808 18:38:10.651173 19555 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:10.651265 19555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:10.651506 19555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:10.653683 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1"
format_stamp: "Formatted at 2026-08-08 18:38:10 on dist-test-slave-fd7l"
server_key: "519b1fc83c8f6c88fd3efdde4c9e15f0"
server_key_iv: "642ca0d4a3093d4d50e1d665bbaa0abb"
server_key_version: "encryptionkey@0"
I20260808 18:38:10.654278 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1"
format_stamp: "Formatted at 2026-08-08 18:38:10 on dist-test-slave-fd7l"
server_key: "519b1fc83c8f6c88fd3efdde4c9e15f0"
server_key_iv: "642ca0d4a3093d4d50e1d665bbaa0abb"
server_key_version: "encryptionkey@0"
I20260808 18:38:10.658114 19555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 18:38:10.660719 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:10.662067 19555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:38:10.662232 19555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1"
format_stamp: "Formatted at 2026-08-08 18:38:10 on dist-test-slave-fd7l"
server_key: "519b1fc83c8f6c88fd3efdde4c9e15f0"
server_key_iv: "642ca0d4a3093d4d50e1d665bbaa0abb"
server_key_version: "encryptionkey@0"
I20260808 18:38:10.662350 19555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:10.688310 19555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:10.691984 19555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:10.692215 19555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:10.699877 19555 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:35989
I20260808 18:38:10.699894 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:35989 every 8 connection(s)
I20260808 18:38:10.700886 19555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:10.703750 19625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:10.704655 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19555
I20260808 18:38:10.704813 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:10.705132 28655 external_mini_cluster.cc:1428] Setting key 7bb135e216a546a2d714d7f466b43fda
I20260808 18:38:10.710273 19625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1: Bootstrap starting.
I20260808 18:38:10.712949 19625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:10.713815 19625 log.cc:824] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:10.714449 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:10.706534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59483 (local address 127.27.251.254:35989)
0808 18:38:10.706949 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:10.706965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:10.707000 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:38:10.707642 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:10.707659 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:10.708024 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:10.708306 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:10.708719 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:10.709694 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:10.710866 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:10.711338 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:10.711455 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:10.712390 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:10.712395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:10.712409 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:10.712883 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:10.712941 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:10.712966 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:10.713136 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:10.713343 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 18:38:10.713511 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260808 18:38:10.716204 19625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1: No bootstrap required, opened a new log
I20260808 18:38:10.717603 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:10.721431 19625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } }
I20260808 18:38:10.721750 19625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:10.721809 19625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4fdc7dda3f4d5cba465fbd8eee86b1, State: Initialized, Role: FOLLOWER
I20260808 18:38:10.722332 19625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } }
I20260808 18:38:10.722498 19625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:10.722572 19625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:10.722678 19625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:10.723991 19625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } }
I20260808 18:38:10.724395 19625 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4fdc7dda3f4d5cba465fbd8eee86b1; no voters:
I20260808 18:38:10.724758 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:10.720033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52057 (local address 127.27.251.254:35989)
0808 18:38:10.720147 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:10.720157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:10.720308 (+ 151us) server_negotiation.cc:407] Connection header received
0808 18:38:10.720424 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:10.720427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:10.720464 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:10.720618 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:10.720928 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:10.721442 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:10.722747 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:10.723047 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:10.723315 (+ 268us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:10.724241 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:10.724244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:10.724246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:10.724258 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:10.724282 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:10.724292 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:10.724362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:10.724622 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 18:38:10.724664 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:38:10.724844 19625 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:10.725152 19635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:10.725409 19635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4fdc7dda3f4d5cba465fbd8eee86b1, State: Running, Role: LEADER
I20260808 18:38:10.725780 19635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } }
I20260808 18:38:10.726187 19625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:10.727572 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4fdc7dda3f4d5cba465fbd8eee86b1. Latest consensus state: current_term: 1 leader_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } } }
I20260808 18:38:10.727591 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4fdc7dda3f4d5cba465fbd8eee86b1" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 35989 } } }
I20260808 18:38:10.727718 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:10.727718 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:10.728139 19648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:10.731389 19648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:10.737104 19648 catalog_manager.cc:1347] Generated new cluster ID: 5f852cbde3d342769aa036a72f56edca
I20260808 18:38:10.737169 19648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:10.750082 19648 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:10.750995 19648 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:10.758289 19648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4fdc7dda3f4d5cba465fbd8eee86b1: Generated new TSK 0
I20260808 18:38:10.758975 19648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:10.764103 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19555
W20260808 18:38:10.771785 19540 connection.cc:535] client connection to 127.27.251.254:35989 recv error: Network error: failed to read from TLS socket (remote: 127.27.251.254:35989): Connection reset by peer (error 104)
2026-08-08T18:38:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:10 dist-test-slave-fd7l krb5kdc[19657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:10 dist-test-slave-fd7l krb5kdc[19657](info): set up 2 sockets
Aug 08 18:38:10 dist-test-slave-fd7l krb5kdc[19657](info): commencing operation
krb5kdc: starting...
W20260808 18:38:11.847144 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:11 dist-test-slave-fd7l krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:11.983002 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37349
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:37349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:12.101181 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:12.101461 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:12.101505 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:12.105566 19670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:12.105628 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:12.105646 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:12.105664 19670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:12.105679 19670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:12.111080 19670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:37349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:37349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:12.112170 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:12.113276 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:12.122543 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:12.122589 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:12.122659 19670 server_base.cc:1034] running on GCE node
W20260808 18:38:12.122584 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:12.123432 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:12.124656 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:12.125823 19670 hybrid_clock.cc:648] HybridClock initialized: now 1786214292125790 us; error 49 us; skew 500 ppm
Aug 08 18:38:12 dist-test-slave-fd7l krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:12.134162 19670 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:12.135298 19670 webserver.cc:460] Webserver started at http://127.27.251.254:34473/ using document root <none> and password file <none>
I20260808 18:38:12.136015 19670 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:12.136121 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:12.136379 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:12.138785 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "332ddbc0fd9045b2beb01e0a1d301814"
format_stamp: "Formatted at 2026-08-08 18:38:12 on dist-test-slave-fd7l"
server_key: "0228c15392d76ada28f58bd79488d54e"
server_key_iv: "6139b3f5c9e61fe1f9ddc9e0e183f43f"
server_key_version: "encryptionkey@0"
I20260808 18:38:12.139430 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "332ddbc0fd9045b2beb01e0a1d301814"
format_stamp: "Formatted at 2026-08-08 18:38:12 on dist-test-slave-fd7l"
server_key: "0228c15392d76ada28f58bd79488d54e"
server_key_iv: "6139b3f5c9e61fe1f9ddc9e0e183f43f"
server_key_version: "encryptionkey@0"
I20260808 18:38:12.143675 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 18:38:12.146174 19685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:12.147426 19670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:38:12.147571 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "332ddbc0fd9045b2beb01e0a1d301814"
format_stamp: "Formatted at 2026-08-08 18:38:12 on dist-test-slave-fd7l"
server_key: "0228c15392d76ada28f58bd79488d54e"
server_key_iv: "6139b3f5c9e61fe1f9ddc9e0e183f43f"
server_key_version: "encryptionkey@0"
I20260808 18:38:12.147706 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:12.152398 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:12.155342 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:12.155536 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:12.163012 19670 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:37349
I20260808 18:38:12.163035 19742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:37349 every 8 connection(s)
I20260808 18:38:12.163944 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:12.167030 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:12.170562 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19670
I20260808 18:38:12.170746 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:12.171100 28655 external_mini_cluster.cc:1428] Setting key 2802eb79b8fd40f002dfa1fdbea2ff64
I20260808 18:38:12.177413 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: Bootstrap starting.
I20260808 18:38:12.178197 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:12.173189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57253 (local address 127.27.251.254:37349)
0808 18:38:12.173769 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:12.173791 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:12.173839 (+ 48us) server_negotiation.cc:407] Connection header received
0808 18:38:12.174829 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:12.174857 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:12.175273 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:12.175695 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:12.176560 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:12.176566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:12.176584 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:12.176597 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:12.176671 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:12.176701 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:12.176854 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:12.176971 (+ 117us) server_negotiation.cc:299] Negotiation successful
0808 18:38:12.177140 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":177,"threads_started":1}
I20260808 18:38:12.180281 19743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:12.181396 19743 log.cc:824] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:12.181831 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:12.184178 19743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: No bootstrap required, opened a new log
I20260808 18:38:12.186806 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:12.184765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58451 (local address 127.27.251.254:37349)
0808 18:38:12.184896 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:12.184901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:12.185079 (+ 178us) server_negotiation.cc:407] Connection header received
0808 18:38:12.185279 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:12.185284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:12.185334 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:12.185431 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:12.186338 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:12.186342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:12.186345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:12.186358 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:12.186403 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:12.186423 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:12.186523 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:12.186624 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 18:38:12.186670 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 18:38:12.189268 19743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } }
I20260808 18:38:12.189584 19743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:12.189653 19743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332ddbc0fd9045b2beb01e0a1d301814, State: Initialized, Role: FOLLOWER
I20260808 18:38:12.190210 19743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } }
I20260808 18:38:12.190415 19743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:12.190487 19743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:12.190594 19743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:12.191998 19743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } }
I20260808 18:38:12.192381 19743 leader_election.cc:302] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332ddbc0fd9045b2beb01e0a1d301814; no voters:
I20260808 18:38:12.192868 19743 leader_election.cc:288] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:12.192992 19753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:12.193236 19753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [term 1 LEADER]: Becoming Leader. State: Replica: 332ddbc0fd9045b2beb01e0a1d301814, State: Running, Role: LEADER
I20260808 18:38:12.193997 19753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } }
I20260808 18:38:12.194496 19743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:12.196043 19754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332ddbc0fd9045b2beb01e0a1d301814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } } }
I20260808 18:38:12.196257 19754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:12.196635 19754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332ddbc0fd9045b2beb01e0a1d301814. Latest consensus state: current_term: 1 leader_uuid: "332ddbc0fd9045b2beb01e0a1d301814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ddbc0fd9045b2beb01e0a1d301814" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 37349 } } }
I20260808 18:38:12.196799 19754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814 [sys.catalog]: This master's current role is: LEADER
W20260808 18:38:12.200819 19768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:12.200908 19768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:12.200985 19764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:12.201860 19764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:12.214257 19764 catalog_manager.cc:1347] Generated new cluster ID: f48443f721614d278f0c528196de288a
I20260808 18:38:12.214423 19764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:12.227556 19764 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:12.227983 19764 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:12.240275 19764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332ddbc0fd9045b2beb01e0a1d301814: Generated new TSK 0
I20260808 18:38:12.241377 19764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:12.246330 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19670
2026-08-08T18:38:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:12 dist-test-slave-fd7l krb5kdc[19774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:12 dist-test-slave-fd7l krb5kdc[19774](info): set up 2 sockets
Aug 08 18:38:12 dist-test-slave-fd7l krb5kdc[19774](info): commencing operation
krb5kdc: starting...
W20260808 18:38:13.327214 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:13.459859 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38011
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:13.575598 19787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:13.575882 19787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:13.575927 19787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:13.580098 19787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:13.580175 19787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:13.580195 19787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:13.580212 19787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:13.580227 19787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:13.585829 19787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:13.586983 19787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:13.588065 19787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:13.597014 19795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:13.597015 19793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:13.597069 19792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:13.597702 19787 server_base.cc:1034] running on GCE node
I20260808 18:38:13.598237 19787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:13.599318 19787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:13.600586 19787 hybrid_clock.cc:648] HybridClock initialized: now 1786214293600553 us; error 41 us; skew 500 ppm
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:13.603694 19787 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:13.604703 19787 webserver.cc:460] Webserver started at http://127.27.251.254:43391/ using document root <none> and password file <none>
I20260808 18:38:13.605297 19787 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:13.605397 19787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:13.605747 19787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:13.608127 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "9de2d91c4fc1458fad71a4bf446de17c"
format_stamp: "Formatted at 2026-08-08 18:38:13 on dist-test-slave-fd7l"
server_key: "ebff6f805455ef2198aa6b0191bf2d5e"
server_key_iv: "7bfdf2a05a0d6377720a9bf756f844c2"
server_key_version: "encryptionkey@0"
I20260808 18:38:13.608824 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "9de2d91c4fc1458fad71a4bf446de17c"
format_stamp: "Formatted at 2026-08-08 18:38:13 on dist-test-slave-fd7l"
server_key: "ebff6f805455ef2198aa6b0191bf2d5e"
server_key_iv: "7bfdf2a05a0d6377720a9bf756f844c2"
server_key_version: "encryptionkey@0"
I20260808 18:38:13.613092 19787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 18:38:13.615621 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:13.617101 19787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 18:38:13.617257 19787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "9de2d91c4fc1458fad71a4bf446de17c"
format_stamp: "Formatted at 2026-08-08 18:38:13 on dist-test-slave-fd7l"
server_key: "ebff6f805455ef2198aa6b0191bf2d5e"
server_key_iv: "7bfdf2a05a0d6377720a9bf756f844c2"
server_key_version: "encryptionkey@0"
I20260808 18:38:13.617370 19787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:13.628894 19787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:13.631719 19787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:13.631927 19787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:13.639182 19787 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:38011
I20260808 18:38:13.639207 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:38011 every 8 connection(s)
I20260808 18:38:13.640131 19787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:13.643016 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:13.648053 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19787
I20260808 18:38:13.648222 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:13.648545 28655 external_mini_cluster.cc:1428] Setting key c1d545aa7e7fc50bb280412bbb950774
I20260808 18:38:13.648882 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c: Bootstrap starting.
I20260808 18:38:13.651384 19860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:13.652140 19860 log.cc:824] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:13.653970 19860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c: No bootstrap required, opened a new log
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:13.658100 19860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } }
I20260808 18:38:13.658306 19860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:13.658365 19860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de2d91c4fc1458fad71a4bf446de17c, State: Initialized, Role: FOLLOWER
I20260808 18:38:13.658846 19860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } }
I20260808 18:38:13.659004 19860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:13.659083 19860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:13.659207 19860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:13.660048 19860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } }
I20260808 18:38:13.660383 19860 leader_election.cc:302] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de2d91c4fc1458fad71a4bf446de17c; no voters:
I20260808 18:38:13.660717 19860 leader_election.cc:288] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:13.660828 19865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:13.661926 19860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:13.663442 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:13.649943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60193 (local address 127.27.251.254:38011)
0808 18:38:13.650385 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:13.650400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:13.650435 (+ 35us) server_negotiation.cc:407] Connection header received
0808 18:38:13.651082 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:13.651099 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:13.651426 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:13.651816 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:13.652681 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:13.653670 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:13.654666 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:13.655148 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:13.655323 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:13.658108 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:13.658114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:13.658127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:13.658153 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:13.660016 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:13.660489 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:13.660494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:13.660498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:13.660668 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:13.661052 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:13.661056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:13.661057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:13.661505 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:13.661864 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:13.662039 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 18:38:13.662366 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":133,"threads_started":1}
I20260808 18:38:13.661067 19865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [term 1 LEADER]: Becoming Leader. State: Replica: 9de2d91c4fc1458fad71a4bf446de17c, State: Running, Role: LEADER
I20260808 18:38:13.664086 19865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } }
I20260808 18:38:13.665685 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de2d91c4fc1458fad71a4bf446de17c. Latest consensus state: current_term: 1 leader_uuid: "9de2d91c4fc1458fad71a4bf446de17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } } }
I20260808 18:38:13.665843 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:13.666697 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:13.666213 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de2d91c4fc1458fad71a4bf446de17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2d91c4fc1458fad71a4bf446de17c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38011 } } }
I20260808 18:38:13.666807 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:13.666350 19880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:13.670336 19880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:13.676550 19880 catalog_manager.cc:1347] Generated new cluster ID: d6ddae573ec5440c83a7acda5be60d56
I20260808 18:38:13.676615 19880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:13.677915 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:13.669336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58697 (local address 127.27.251.254:38011)
0808 18:38:13.669443 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:13.669448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:13.669462 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:38:13.669663 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:13.669668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:13.669719 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:13.669808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:13.670685 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:13.671487 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:13.672562 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:13.672810 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:13.672931 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:13.674347 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:13.674351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:13.674353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:13.674379 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:13.676148 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:13.676669 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:13.676673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:13.676674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:13.676766 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:13.677041 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:13.677044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:13.677046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:13.677233 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:13.677369 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:13.677612 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 18:38:13.677740 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 18:38:13.685760 19880 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:13.686739 19880 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:13.691869 19880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de2d91c4fc1458fad71a4bf446de17c: Generated new TSK 0
I20260808 18:38:13.692682 19880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:13.698158 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19787
2026-08-08T18:38:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19892](info): set up 2 sockets
Aug 08 18:38:13 dist-test-slave-fd7l krb5kdc[19892](info): commencing operation
krb5kdc: starting...
W20260808 18:38:14.803800 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:14 dist-test-slave-fd7l krb5kdc[19892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:14.935866 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34101
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:34101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:15.052055 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:15.052350 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:15.052393 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:15.056308 19905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:15.056371 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:15.056391 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:15.056407 19905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:15.056422 19905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:15.061748 19905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:34101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:34101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:15.062891 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:15.064232 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:15.073729 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:15.073719 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:15.073700 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:15.073832 19905 server_base.cc:1034] running on GCE node
I20260808 18:38:15.074486 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:15.075542 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:15.076701 19905 hybrid_clock.cc:648] HybridClock initialized: now 1786214295076682 us; error 32 us; skew 500 ppm
Aug 08 18:38:15 dist-test-slave-fd7l krb5kdc[19892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:15.079488 19905 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:15.080400 19905 webserver.cc:460] Webserver started at http://127.27.251.254:44341/ using document root <none> and password file <none>
I20260808 18:38:15.080979 19905 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:15.081041 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:15.081283 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:15.083410 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "a3d487d26e09447689a2fea02b408d9b"
format_stamp: "Formatted at 2026-08-08 18:38:15 on dist-test-slave-fd7l"
server_key: "e66f1e0ead3eae8dc688b4a37aa02852"
server_key_iv: "8599a47e1024a78cffc76401b03f8dbd"
server_key_version: "encryptionkey@0"
I20260808 18:38:15.083969 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "a3d487d26e09447689a2fea02b408d9b"
format_stamp: "Formatted at 2026-08-08 18:38:15 on dist-test-slave-fd7l"
server_key: "e66f1e0ead3eae8dc688b4a37aa02852"
server_key_iv: "8599a47e1024a78cffc76401b03f8dbd"
server_key_version: "encryptionkey@0"
I20260808 18:38:15.087740 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260808 18:38:15.090358 19923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:15.091661 19905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 18:38:15.091807 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "a3d487d26e09447689a2fea02b408d9b"
format_stamp: "Formatted at 2026-08-08 18:38:15 on dist-test-slave-fd7l"
server_key: "e66f1e0ead3eae8dc688b4a37aa02852"
server_key_iv: "8599a47e1024a78cffc76401b03f8dbd"
server_key_version: "encryptionkey@0"
I20260808 18:38:15.091922 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:15.104130 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:15.107246 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:15.107457 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:15.115154 19905 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:34101
I20260808 18:38:15.115173 19978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:34101 every 8 connection(s)
I20260808 18:38:15.116108 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:15.118891 19979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:15.123653 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 19905
I20260808 18:38:15.123803 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:15.124092 28655 external_mini_cluster.cc:1428] Setting key cc453424871484a7eca29e89508a0278
I20260808 18:38:15.125016 19979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: Bootstrap starting.
I20260808 18:38:15.127470 19979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:15.128247 19979 log.cc:824] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:15.130187 19979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: No bootstrap required, opened a new log
I20260808 18:38:15.133231 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:15.125584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54997 (local address 127.27.251.254:34101)
0808 18:38:15.126021 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:15.126036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:15.126072 (+ 36us) server_negotiation.cc:407] Connection header received
0808 18:38:15.126723 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:15.126740 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:15.127079 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:15.127425 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:15.127847 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:15.128938 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:15.129851 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:15.130380 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:15.130571 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:15.131444 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:15.131448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:15.131462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:15.131881 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:15.131937 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:15.131961 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:15.132074 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:15.132236 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 18:38:15.132443 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260808 18:38:15.134092 19979 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } }
I20260808 18:38:15.134390 19979 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:15.134451 19979 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d487d26e09447689a2fea02b408d9b, State: Initialized, Role: FOLLOWER
I20260808 18:38:15.134904 19979 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } }
I20260808 18:38:15.135092 19979 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:15.135166 19979 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:15.135313 19979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:15.136197 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:15.136442 19979 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } }
I20260808 18:38:15.136818 19979 leader_election.cc:302] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3d487d26e09447689a2fea02b408d9b; no voters:
I20260808 18:38:15.137177 19979 leader_election.cc:288] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:15.137274 19984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:15.137488 19984 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [term 1 LEADER]: Becoming Leader. State: Replica: a3d487d26e09447689a2fea02b408d9b, State: Running, Role: LEADER
I20260808 18:38:15.137840 19984 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } }
I20260808 18:38:15.138520 19979 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:15.139451 19985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3d487d26e09447689a2fea02b408d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } } }
I20260808 18:38:15.139616 19985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:15.139984 19986 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3d487d26e09447689a2fea02b408d9b. Latest consensus state: current_term: 1 leader_uuid: "a3d487d26e09447689a2fea02b408d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d487d26e09447689a2fea02b408d9b" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 34101 } } }
I20260808 18:38:15.140129 19986 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:15.143477 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:15.138480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55301 (local address 127.27.251.254:34101)
0808 18:38:15.138600 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:15.138604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:15.139315 (+ 711us) server_negotiation.cc:407] Connection header received
0808 18:38:15.139515 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:15.139518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:15.139564 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:15.139619 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:15.140105 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:15.140864 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:15.141853 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:15.142100 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:15.142229 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:15.143042 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:15.143046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:15.143049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:15.143061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:15.143095 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:15.143107 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:15.143184 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:15.143276 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 18:38:15.143325 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 18:38:15.143770 20007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:15.144340 20007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:15.144430 19999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:15.145346 19999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:15.151310 19999 catalog_manager.cc:1347] Generated new cluster ID: 810c9fff48d6476d876179081e4b37b4
I20260808 18:38:15.151371 19999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:15.161123 19999 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:15.162024 19999 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:15.168253 19999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3d487d26e09447689a2fea02b408d9b: Generated new TSK 0
I20260808 18:38:15.168951 19999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:15.175289 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 19905
2026-08-08T18:38:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:15 dist-test-slave-fd7l krb5kdc[20012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:15 dist-test-slave-fd7l krb5kdc[20012](info): set up 2 sockets
Aug 08 18:38:15 dist-test-slave-fd7l krb5kdc[20012](info): commencing operation
krb5kdc: starting...
W20260808 18:38:16.295059 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:16 dist-test-slave-fd7l krb5kdc[20012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:16.429286 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:39543
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:16.545066 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:16.545333 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:16.545375 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:16.549510 20027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:16.549567 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:16.549587 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:16.549611 20027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:16.549656 20027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:16.554943 20027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:16.556030 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:16.557111 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:16.565994 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:16.566028 20027 server_base.cc:1034] running on GCE node
W20260808 18:38:16.566056 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:16.566033 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:16.566800 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:16.567929 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:16.569120 20027 hybrid_clock.cc:648] HybridClock initialized: now 1786214296569091 us; error 42 us; skew 500 ppm
Aug 08 18:38:16 dist-test-slave-fd7l krb5kdc[20012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:16.571861 20027 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:16.572885 20027 webserver.cc:460] Webserver started at http://127.27.251.254:40935/ using document root <none> and password file <none>
I20260808 18:38:16.573472 20027 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:16.573565 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:16.573808 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:16.575883 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "ea9ff94bcfda42d9a406567cb52ff81c"
format_stamp: "Formatted at 2026-08-08 18:38:16 on dist-test-slave-fd7l"
server_key: "e6894b2c9f7570a1bb60a5cf1998ecbf"
server_key_iv: "d0dabf7504593d3e154ff55c98932360"
server_key_version: "encryptionkey@0"
I20260808 18:38:16.576483 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "ea9ff94bcfda42d9a406567cb52ff81c"
format_stamp: "Formatted at 2026-08-08 18:38:16 on dist-test-slave-fd7l"
server_key: "e6894b2c9f7570a1bb60a5cf1998ecbf"
server_key_iv: "d0dabf7504593d3e154ff55c98932360"
server_key_version: "encryptionkey@0"
I20260808 18:38:16.580217 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 18:38:16.582739 20044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:16.584044 20027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:38:16.584198 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "ea9ff94bcfda42d9a406567cb52ff81c"
format_stamp: "Formatted at 2026-08-08 18:38:16 on dist-test-slave-fd7l"
server_key: "e6894b2c9f7570a1bb60a5cf1998ecbf"
server_key_iv: "d0dabf7504593d3e154ff55c98932360"
server_key_version: "encryptionkey@0"
I20260808 18:38:16.584321 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:16.589031 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:16.592084 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:16.592291 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:16.600452 20027 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:39543
I20260808 18:38:16.600528 20100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:39543 every 8 connection(s)
I20260808 18:38:16.601485 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:16.604439 20101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:16.606680 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20027
I20260808 18:38:16.606833 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:16.607128 28655 external_mini_cluster.cc:1428] Setting key cca36106b55f5a8b914a8fe533b2c695
I20260808 18:38:16.611691 20101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c: Bootstrap starting.
I20260808 18:38:16.612720 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:16.608566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46253 (local address 127.27.251.254:39543)
0808 18:38:16.608976 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:16.608992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:16.609047 (+ 55us) server_negotiation.cc:407] Connection header received
0808 18:38:16.609729 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:16.609753 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:16.610125 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:16.610493 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:16.611230 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:16.611238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:16.611253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:16.611266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:16.611330 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:16.611357 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:16.611498 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:16.611644 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 18:38:16.611803 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":129,"threads_started":1}
I20260808 18:38:16.614384 20101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:16.615252 20101 log.cc:824] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:16.615679 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:16.617400 20101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c: No bootstrap required, opened a new log
I20260808 18:38:16.619606 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:16.617909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40425 (local address 127.27.251.254:39543)
0808 18:38:16.618028 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:16.618032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:16.618196 (+ 164us) server_negotiation.cc:407] Connection header received
0808 18:38:16.618325 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:16.618327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:16.618356 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:16.618414 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:16.619142 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:16.619144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:16.619146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 18:38:16.619157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:16.619183 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 18:38:16.619193 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:16.619276 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:16.619383 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 18:38:16.619413 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:38:16.621734 20101 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } }
I20260808 18:38:16.621953 20101 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 18:38:16.621986 20101 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9ff94bcfda42d9a406567cb52ff81c, State: Initialized, Role: FOLLOWER
I20260808 18:38:16.622469 20101 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } }
I20260808 18:38:16.622598 20101 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 18:38:16.622675 20101 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 18:38:16.622793 20101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:16.623714 20101 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } }
I20260808 18:38:16.624064 20101 leader_election.cc:302] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9ff94bcfda42d9a406567cb52ff81c; no voters:
I20260808 18:38:16.624398 20101 leader_election.cc:288] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 18:38:16.624634 20111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:16.624920 20111 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [term 1 LEADER]: Becoming Leader. State: Replica: ea9ff94bcfda42d9a406567cb52ff81c, State: Running, Role: LEADER
I20260808 18:38:16.625300 20111 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } }
I20260808 18:38:16.625862 20101 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:16.627079 20112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } } }
I20260808 18:38:16.627236 20112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:16.627615 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9ff94bcfda42d9a406567cb52ff81c. Latest consensus state: current_term: 1 leader_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9ff94bcfda42d9a406567cb52ff81c" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 39543 } } }
I20260808 18:38:16.627686 20120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:16.627714 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:16.630597 20120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:16.636078 20120 catalog_manager.cc:1347] Generated new cluster ID: cbb2982782674503bf3c2380e12eba5f
I20260808 18:38:16.636152 20120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:16.661892 20120 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:16.662289 20120 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:16.670152 20120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9ff94bcfda42d9a406567cb52ff81c: Generated new TSK 0
I20260808 18:38:16.670981 20120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 18:38:16.681844 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20027
2026-08-08T18:38:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1506 ms)
[----------] 18 tests from AuthTokenIssuingTest (26897 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:16 dist-test-slave-fd7l krb5kdc[20132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:16 dist-test-slave-fd7l krb5kdc[20132](info): set up 2 sockets
Aug 08 18:38:16 dist-test-slave-fd7l krb5kdc[20132](info): commencing operation
krb5kdc: starting...
W20260808 18:38:17.778838 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:17 dist-test-slave-fd7l krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:17.911548 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38551
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:18.031373 20145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:18.031746 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:18.031822 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:18.031905 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:18.036069 20145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:18.036171 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:18.036209 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:18.036273 20145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:18.036312 20145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:18.041846 20145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:18.043047 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:18.044190 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:18.053233 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:18.053304 20145 server_base.cc:1034] running on GCE node
W20260808 18:38:18.053227 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:18.053225 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:18.054185 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:18.055403 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:18.056584 20145 hybrid_clock.cc:648] HybridClock initialized: now 1786214298056543 us; error 48 us; skew 500 ppm
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:18.064765 20145 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:18.065796 20145 webserver.cc:460] Webserver started at http://127.27.251.254:36335/ using document root <none> and password file <none>
I20260808 18:38:18.066431 20145 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:18.066530 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:18.066780 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:18.069096 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "52bc830f3c664488b6848d59bd145500"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "1fecf467b96e296b1d220824603885d2"
server_key_iv: "952dc5835e03d68b654be41d636f8f46"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.069712 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "52bc830f3c664488b6848d59bd145500"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "1fecf467b96e296b1d220824603885d2"
server_key_iv: "952dc5835e03d68b654be41d636f8f46"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.074046 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 18:38:18.077251 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:18.079172 20145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:38:18.079280 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "52bc830f3c664488b6848d59bd145500"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "1fecf467b96e296b1d220824603885d2"
server_key_iv: "952dc5835e03d68b654be41d636f8f46"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.079371 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:18.106050 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:18.109256 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:18.109454 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:18.117133 20145 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:38551
I20260808 18:38:18.117486 20216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:38551 every 8 connection(s)
I20260808 18:38:18.118181 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:18.119848 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20145
I20260808 18:38:18.119997 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:18.120292 28655 external_mini_cluster.cc:1428] Setting key 35c6de4d934403413708220e4a12aff8
I20260808 18:38:18.121460 20217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:18.127396 20217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.253@KRBTEST.COM" created.
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:18.144917 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.130781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.27.251.254:38551)
0808 18:38:18.131433 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.131452 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.131693 (+ 241us) server_negotiation.cc:407] Connection header received
0808 18:38:18.132954 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.132959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.133163 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.133326 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.134374 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.135272 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.136350 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.136889 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.137034 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.139709 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.139715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.139726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.139748 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.141570 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.142065 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.142078 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.142080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.142214 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.142541 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.142546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.142548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.142938 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.143092 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.143471 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.143766 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":521,"thread_start_us":188,"threads_started":1}
I20260808 18:38:18.144917 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.130781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38551 (local address 127.0.0.1:52954)
0808 18:38:18.131475 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.131559 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.132652 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.133362 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.133367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.133810 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.134197 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.134224 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.135414 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.135417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.136181 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.136197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.137069 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.137077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.137285 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.137984 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.138024 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.139524 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.141689 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.141700 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.141711 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.141950 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.142326 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.142329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.142331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.142446 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.143060 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.143067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.143218 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.143395 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.143818 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":160,"spinlock_wait_cycles":43264,"thread_start_us":188,"threads_started":1}
I20260808 18:38:18.147641 20217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.148437 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.148144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:44994)
0808 18:38:18.148266 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.148341 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:38:18.148705 20164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
I20260808 18:38:18.149660 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.149340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45008)
0808 18:38:18.149553 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.149601 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 18:38:18.149824 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:43623: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
I20260808 18:38:18.152132 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:43623
--webserver_interface=127.27.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:38:18.199107 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } attempt: 1
I20260808 18:38:18.200057 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.199745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45024)
0808 18:38:18.199900 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.199976 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 18:38:18.200968 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.200739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45036)
0808 18:38:18.200859 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.200912 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:38:18.201144 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:43623: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
I20260808 18:38:18.269353 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } attempt: 2
I20260808 18:38:18.270318 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.269983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45046)
0808 18:38:18.270149 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.270238 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 18:38:18.271059 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.270857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45054)
0808 18:38:18.270957 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.271001 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:38:18.271200 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:43623: Network error: Client connection negotiation failed: client connection to 127.27.251.253:43623: connect: Connection refused (error 111)
W20260808 18:38:18.274780 20224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:18.275064 20224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:18.275134 20224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:18.275223 20224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:18.279269 20224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:18.279322 20224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:18.279341 20224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:18.279403 20224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:18.279428 20224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:18.285156 20224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:18.286248 20224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:18.287338 20224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:18.296434 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:18.296643 20230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:18.296481 20224 server_base.cc:1034] running on GCE node
W20260808 18:38:18.296458 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:18.297375 20224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:18.298565 20224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:18.299765 20224 hybrid_clock.cc:648] HybridClock initialized: now 1786214298299748 us; error 28 us; skew 500 ppm
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:18.302873 20224 init.cc:377] Logged in from keytab as kudu/127.27.251.253@KRBTEST.COM (short username kudu)
I20260808 18:38:18.303990 20224 webserver.cc:460] Webserver started at http://127.27.251.253:33561/ using document root <none> and password file <none>
I20260808 18:38:18.304669 20224 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:18.304745 20224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:18.304940 20224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:18.307245 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data/instance:
uuid: "fd54916fdf834c7598d517beb55b9b8a"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "f7ec49f144a71e4edeb241af7baf0095"
server_key_iv: "a897b95560ff5b4393a3a46676d6b2eb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.307893 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal/instance:
uuid: "fd54916fdf834c7598d517beb55b9b8a"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "f7ec49f144a71e4edeb241af7baf0095"
server_key_iv: "a897b95560ff5b4393a3a46676d6b2eb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.311939 20224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:38:18.314499 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:18.315834 20224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 18:38:18.315994 20224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "fd54916fdf834c7598d517beb55b9b8a"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "f7ec49f144a71e4edeb241af7baf0095"
server_key_iv: "a897b95560ff5b4393a3a46676d6b2eb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.316120 20224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:18.332585 20224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:18.336052 20224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:18.336273 20224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:18.343773 20224 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.253:43623
I20260808 18:38:18.344072 20294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.253:43623 every 8 connection(s)
I20260808 18:38:18.344828 20224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
I20260808 18:38:18.347726 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:18.350255 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20224
I20260808 18:38:18.350392 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-1/wal/instance
I20260808 18:38:18.350708 28655 external_mini_cluster.cc:1428] Setting key ddc663db6e8d3464f4986b8551852abf
I20260808 18:38:18.352407 20295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.251.252@KRBTEST.COM; defaulting to no policy
I20260808 18:38:18.362375 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.252@KRBTEST.COM" created.
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:38:18.369820 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.354676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.27.251.254:38551)
0808 18:38:18.354836 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.354841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.355731 (+ 890us) server_negotiation.cc:407] Connection header received
0808 18:38:18.356726 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.356732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.356808 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.356932 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.358136 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.359003 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.360176 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.360567 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.360667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.364004 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.364010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.364014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.364041 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.367182 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.367756 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.367762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.367767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.367863 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.368287 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.368291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.368293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.368487 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.368734 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.369480 (+ 746us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.369605 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 18:38:18.371076 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.355071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38551 (local address 127.0.0.1:52958)
0808 18:38:18.355549 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.355604 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.356449 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.357075 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.357082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.357417 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.357949 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.357964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.359135 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.359138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.360009 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.360018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.360747 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.360750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.360978 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.361577 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.361594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.363830 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.367305 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.367311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.367329 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.367585 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.367983 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.367986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.367988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.368194 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.368762 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.368770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.369146 (+ 376us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.369415 (+ 269us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.369865 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":139,"threads_started":1}
I20260808 18:38:18.372783 20295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.373853 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.362910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45056)
0808 18:38:18.363054 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.363071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.363163 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.363956 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.363961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.364179 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.364320 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.364333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.365843 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.365849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.366368 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.366376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.367245 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.367253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.367419 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.367948 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.367971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.369432 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.371642 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.371645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.371647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.371890 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.372314 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.372317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.372319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.372413 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.373085 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.373088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.373212 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.373322 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.373743 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 18:38:18.374426 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.363033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.27.251.253:43623)
0808 18:38:18.363369 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.363385 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.363426 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:38:18.363491 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.363497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.363746 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.363955 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.364443 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.365723 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.366497 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.367071 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.367205 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.369589 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.369600 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.369614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.369645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.371516 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.372025 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.372035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.372040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.372145 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.372548 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.372554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.372559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.372937 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.373090 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.374097 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.374297 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":133,"threads_started":1}
I20260808 18:38:18.376216 20217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.376983 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.376701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46822)
0808 18:38:18.376811 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.376866 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 18:38:18.377897 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.377621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46836)
0808 18:38:18.377761 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.377805 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260808 18:38:18.378047 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
I20260808 18:38:18.382898 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.373321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45072)
0808 18:38:18.373489 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.373506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.373581 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.374017 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.374021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.374200 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.374330 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.374337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.375312 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.375316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.376129 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.376141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.376779 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.376783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.376946 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.377410 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.377428 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.378823 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.381384 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.381387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.381388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.381627 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.381865 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.381867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.381868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.381953 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.382411 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.382417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.382587 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.382697 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.382794 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":50}
I20260808 18:38:18.383116 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.373394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.27.251.253:43623)
0808 18:38:18.373755 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.373763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.373776 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:38:18.373806 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.373808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.373842 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.374425 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.374526 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.375185 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.376264 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.376662 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.376781 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.379018 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.379022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.379024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.379045 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.381281 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.381708 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.381710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.381711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.381784 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.382069 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.382072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.382073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.382313 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.382411 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.382782 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.382907 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260808 18:38:18.383460 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:38855
--webserver_interface=127.27.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:38:18.383917 20295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.384690 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.384371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46850)
0808 18:38:18.384485 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.384639 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:38:18.384904 20244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
I20260808 18:38:18.385795 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.385589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46858)
0808 18:38:18.385706 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.385749 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 18:38:18.386018 20295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
I20260808 18:38:18.409200 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 1
I20260808 18:38:18.410140 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.409794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46872)
0808 18:38:18.409989 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.410053 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 18:38:18.410988 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.410778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46880)
0808 18:38:18.410889 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.410934 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:38:18.411198 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
I20260808 18:38:18.435293 20295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 1
I20260808 18:38:18.436352 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.436000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46886)
0808 18:38:18.436193 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.436263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 18:38:18.437179 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.436950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46898)
0808 18:38:18.437062 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.437114 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:38:18.437320 20295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
I20260808 18:38:18.486359 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 2
I20260808 18:38:18.487272 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.486956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46908)
0808 18:38:18.487122 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.487192 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 18:38:18.488035 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.487820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46922)
0808 18:38:18.487930 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.487983 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:38:18.488169 20217 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
W20260808 18:38:18.503935 20302 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:18.504276 20302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:18.504343 20302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:18.504442 20302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 18:38:18.505487 20295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 2
I20260808 18:38:18.506299 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.505980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46930)
0808 18:38:18.506156 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.506221 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 18:38:18.506995 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.506787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46946)
0808 18:38:18.506897 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.506946 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 18:38:18.507133 20295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:38855: Network error: Client connection negotiation failed: client connection to 127.27.251.252:38855: connect: Connection refused (error 111)
W20260808 18:38:18.508770 20302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:18.508846 20302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:18.508898 20302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:18.508942 20302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:18.508982 20302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:18.514386 20302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:38551,127.27.251.253:43623,127.27.251.252:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:18.515511 20302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:18.516645 20302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:18.525398 20309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:18.525349 20308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:18.525379 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:18.525609 20302 server_base.cc:1034] running on GCE node
I20260808 18:38:18.526221 20302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:18.527453 20302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:18.528648 20302 hybrid_clock.cc:648] HybridClock initialized: now 1786214298528612 us; error 44 us; skew 500 ppm
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:18.531934 20302 init.cc:377] Logged in from keytab as kudu/127.27.251.252@KRBTEST.COM (short username kudu)
I20260808 18:38:18.532852 20302 webserver.cc:460] Webserver started at http://127.27.251.252:36795/ using document root <none> and password file <none>
I20260808 18:38:18.533386 20302 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:18.533447 20302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:18.533613 20302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:18.535678 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data/instance:
uuid: "27786d89e5674e0d859aaeb90f4d71ec"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "892a67462738639d89072f6bf150b590"
server_key_iv: "76ce71a6dd774de179ee317d4b7fddeb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.536247 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal/instance:
uuid: "27786d89e5674e0d859aaeb90f4d71ec"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "892a67462738639d89072f6bf150b590"
server_key_iv: "76ce71a6dd774de179ee317d4b7fddeb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.540146 20302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 18:38:18.542814 20320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:18.544094 20302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:38:18.544256 20302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "27786d89e5674e0d859aaeb90f4d71ec"
format_stamp: "Formatted at 2026-08-08 18:38:18 on dist-test-slave-fd7l"
server_key: "892a67462738639d89072f6bf150b590"
server_key_iv: "76ce71a6dd774de179ee317d4b7fddeb"
server_key_version: "encryptionkey@0"
I20260808 18:38:18.544378 20302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:18.564183 20302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:18.567670 20302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:18.567884 20302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:18.574993 20302 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.252:38855
I20260808 18:38:18.575312 20374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.252:38855 every 8 connection(s)
I20260808 18:38:18.575999 20302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
I20260808 18:38:18.578814 20375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:18.581164 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20302
I20260808 18:38:18.581331 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786214007390461-28655-0/minicluster-data/master-2/wal/instance
I20260808 18:38:18.581707 28655 external_mini_cluster.cc:1428] Setting key a3004d6c0d1249b7a32d0541db7a9fba
I20260808 18:38:18.583710 20375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.587328 20217 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 3
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:38:18.598281 20295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } attempt: 3
I20260808 18:38:18.598688 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.586644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.27.251.254:38551)
0808 18:38:18.586813 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.586818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.587092 (+ 274us) server_negotiation.cc:407] Connection header received
0808 18:38:18.587223 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.587226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.587280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.587376 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.588372 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.588995 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.590349 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.590843 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.590950 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.594523 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.594528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.594531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.594558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.596451 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597133 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.597137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.597139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.597235 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597558 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.597562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.597564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.597819 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.597890 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.598339 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.598497 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":72}
I20260808 18:38:18.598989 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.587768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46964)
0808 18:38:18.587899 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.587914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.587995 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.588584 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.588587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.588807 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.588907 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.588914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.589764 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.589768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.590707 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.590715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.591391 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.591397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.591819 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.592631 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.592648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.594186 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.597189 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.597197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.597478 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.597825 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.597831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.597939 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.598487 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.598491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.598582 (+ 91us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.598668 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.598822 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":60}
I20260808 18:38:18.599906 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.583200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.27.251.252:38855)
0808 18:38:18.583674 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.583697 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.583747 (+ 50us) server_negotiation.cc:407] Connection header received
0808 18:38:18.584770 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.584799 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.585253 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.585653 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.586525 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.588218 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.589560 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.590115 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.590296 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.593350 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.593360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.593375 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.593406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.595450 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.596203 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.596213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.596222 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.596356 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.596763 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.596768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.596770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.597366 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.597565 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.597952 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.598633 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
I20260808 18:38:18.599895 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.586550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:38551 (local address 127.0.0.1:52962)
0808 18:38:18.586942 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.586991 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.587112 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.587448 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.587454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.587983 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.588202 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.588213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.589131 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.589134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.590205 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.590217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.591006 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.591010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.591171 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.591738 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.591756 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.594335 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.596609 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.596618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.596625 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.597013 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.597346 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.597351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.597447 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.597972 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.597978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.598083 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.598269 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.599314 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":839,"thread_start_us":143,"threads_started":1}
I20260808 18:38:18.599884 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.588013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.27.251.252:38855)
0808 18:38:18.588321 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.588332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.588346 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:38:18.588378 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.588380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.588419 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.588545 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.589041 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.589643 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.590834 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.591249 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.591378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.594356 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.594362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.594367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.594396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.597055 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.597601 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.597605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.597606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.597700 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.598094 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.598097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.598099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.598358 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.599266 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.599352 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.599472 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260808 18:38:18.601156 20375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.605023 20217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: Bootstrap starting.
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:38:18.608135 20217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:18.609239 20217 log.cc:824] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: Log is configured to *not* fsync() on all Append() calls
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:18.612421 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.602236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.27.251.253:43623)
0808 18:38:18.602388 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.602394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.602410 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:38:18.602493 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.602496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.602548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.602646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.603088 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.603810 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.604903 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.605283 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.605352 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.608262 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.608268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.608273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.608306 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.609974 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.610562 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.610567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.610570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.610666 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.611317 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.611322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.611327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.611644 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.611723 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.612103 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.612235 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:38:18.612582 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.598985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.27.251.252:38855)
0808 18:38:18.601139 (+ 2154us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.601145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.601159 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:38:18.601193 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.601196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.601234 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.601451 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.601972 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.602678 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.603780 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.604765 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.606559 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.607981 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.607988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.607994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.608048 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.610394 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.611046 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.611052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.611054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.611148 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.611521 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.611524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.611527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.611743 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.612200 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.612276 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.612379 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":624,"server-negotiator.queue_time_us":2077,"spinlock_wait_cycles":12800,"thread_start_us":112,"threads_started":1}
I20260808 18:38:18.612715 20217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: No bootstrap required, opened a new log
I20260808 18:38:18.613476 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.601658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:43623 (local address 127.0.0.1:45088)
0808 18:38:18.601775 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.601790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.601857 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.602690 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.602693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.602863 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.602960 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.602966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.603918 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.603921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.604781 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.604792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.605500 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.605507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.605603 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.606030 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.606044 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.608067 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.610092 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.610097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.610099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.610455 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.610802 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.610806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.610809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.611201 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.611807 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.611811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.611907 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.613094 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.613262 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3,"spinlock_wait_cycles":1280}
I20260808 18:38:18.613835 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.603377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.27.251.254:38551)
0808 18:38:18.603504 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.603511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.603813 (+ 302us) server_negotiation.cc:407] Connection header received
0808 18:38:18.603858 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.603863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.603909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.603991 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.604753 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.605442 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.606579 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.606944 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.607042 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.609960 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.609963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.609966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.609989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.611779 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.612258 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.612262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.612263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.612348 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.612858 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.612864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.612867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.613097 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.613294 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.613532 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.613671 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 18:38:18.614336 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.598753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46980)
0808 18:38:18.599041 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.599060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.599152 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.601481 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.601485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.601720 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.601847 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.601857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.602806 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.602812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.603641 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.603651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.604923 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.604927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.605021 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.605627 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.605644 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.607807 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.610529 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.610533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.610537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.610870 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.611279 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.611285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.611291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.611396 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.611864 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.611868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.611978 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.612085 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.613876 (+ 1791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1630}
I20260808 18:38:18.615340 20375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:18.617895 20295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: Bootstrap starting.
I20260808 18:38:18.618716 20217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.618958 20217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bc830f3c664488b6848d59bd145500, State: Initialized, Role: FOLLOWER
I20260808 18:38:18.619405 20217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.621030 20295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: Neither blocks nor log segments found. Creating new log.
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:38:18.621177 20217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 18:38:18 dist-test-slave-fd7l krb5kdc[20132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:38:18.621841 20295 log.cc:824] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:18.623624 20383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.624754 20383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.625975 20295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: No bootstrap required, opened a new log
I20260808 18:38:18.626554 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.615809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:38855 (local address 127.0.0.1:46994)
0808 18:38:18.615902 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:18.615917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:18.615996 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:18.616243 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:18.616246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:18.616416 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:18.616491 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.616496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.617605 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.617609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.618468 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:18.618479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.619132 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.619136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:18.619217 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.619815 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:18.619831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:18.622399 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:18.624708 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.624711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.624715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.624992 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.625314 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:18.625317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:18.625320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:18.625408 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:18.625982 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:18.625985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:18.626078 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 18:38:18.626190 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 18:38:18.626347 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 18:38:18.626755 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.615697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.27.251.253:43623)
0808 18:38:18.615814 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.615818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.616087 (+ 269us) server_negotiation.cc:407] Connection header received
0808 18:38:18.616251 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.616254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.616294 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.616340 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.617225 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.617908 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.618919 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.619255 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.619354 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.622480 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.622487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.622493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.622521 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.624688 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.625219 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.625222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.625224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.625314 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.625661 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.625666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.625667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.625897 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.626042 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.626440 (+ 398us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.626572 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 18:38:18.627177 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.615901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.27.251.252:38855)
0808 18:38:18.615992 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.615996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.616010 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:38:18.616089 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.616094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.616131 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.616230 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.616674 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.617483 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.618609 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.618996 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.622558 (+ 3562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.622687 (+ 129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.622693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.622696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.622715 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.624580 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.625103 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.625109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.625111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.625202 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.625519 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.625522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.625524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.625793 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.625930 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.626889 (+ 959us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.627016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 18:38:18.630427 20295 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.630609 20375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: Bootstrap starting.
I20260808 18:38:18.630733 20295 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd54916fdf834c7598d517beb55b9b8a, State: Initialized, Role: FOLLOWER
I20260808 18:38:18.631060 20295 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.627382 20145 master_runner.cc:186] Error getting master registration for 127.27.251.253:43623: OK, instance_id { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" instance_seqno: 1786214298341661 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:38:18.631785 20145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 18:38:18.631983 20395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:18.632045 20395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:18.632557 20295 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:18.633571 20375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:18.634580 20375 log.cc:824] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:18.635509 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:18.636910 20191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "52bc830f3c664488b6848d59bd145500"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52974
I20260808 18:38:18.637131 20191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 18:38:18.637236 20191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:18.637390 20375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: No bootstrap required, opened a new log
I20260808 18:38:18.638576 20191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.639591 20191 leader_election.cc:288] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [CANDIDATE]: Term 1 election: Requested vote from peers fd54916fdf834c7598d517beb55b9b8a (127.27.251.253:43623), 27786d89e5674e0d859aaeb90f4d71ec (127.27.251.252:38855)
I20260808 18:38:18.641397 20267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52bc830f3c664488b6848d59bd145500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd54916fdf834c7598d517beb55b9b8a"
I20260808 18:38:18.641772 20267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:18.641677 20375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.641927 20375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27786d89e5674e0d859aaeb90f4d71ec, State: Initialized, Role: FOLLOWER
I20260808 18:38:18.642326 20375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.643389 20267 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bc830f3c664488b6848d59bd145500 in term 1.
I20260808 18:38:18.643810 20375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:38:18.644374 20409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:18.644479 20409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:18.644688 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.644853 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.645537 20348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52bc830f3c664488b6848d59bd145500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27786d89e5674e0d859aaeb90f4d71ec"
I20260808 18:38:18.645804 20348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:18.647183 20348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52bc830f3c664488b6848d59bd145500 in term 1.
I20260808 18:38:18.650996 20164 leader_election.cc:302] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52bc830f3c664488b6848d59bd145500, fd54916fdf834c7598d517beb55b9b8a; no voters:
I20260808 18:38:18.651340 20383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:18.651573 20383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [term 1 LEADER]: Becoming Leader. State: Replica: 52bc830f3c664488b6848d59bd145500, State: Running, Role: LEADER
I20260808 18:38:18.651957 20383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } }
I20260808 18:38:18.652685 20414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
W20260808 18:38:18.652767 20431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 18:38:18.652809 20414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: This master's current role is: FOLLOWER
W20260808 18:38:18.652837 20431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:18.654090 20383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bc830f3c664488b6848d59bd145500. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.654209 20383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:18.659572 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.648723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.27.251.253:43623)
0808 18:38:18.649354 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.649359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.649371 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:38:18.649407 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.649410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.649454 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.649528 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.650816 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.651546 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.652787 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.653103 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.653224 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.655179 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.655183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.655186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.655215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.657250 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.657811 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.657815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.657817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.657907 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.658263 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.658270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.658273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.658512 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.658644 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.659275 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.659404 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533}
I20260808 18:38:18.661666 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.646426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.27.251.252:38855)
0808 18:38:18.646545 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.646552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.646922 (+ 370us) server_negotiation.cc:407] Connection header received
0808 18:38:18.646994 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.647001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.647067 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.647187 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.648274 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.649169 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.654459 (+ 5290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.654793 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.654870 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.657103 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.657107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.657110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.657137 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.658905 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.659419 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.659423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.659425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.659511 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.659829 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.659832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.659834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.660049 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.660158 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.661377 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.661501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 18:38:18.662173 20224 master_runner.cc:263] No leader master found from master fd54916fdf834c7598d517beb55b9b8a
I20260808 18:38:18.662249 20224 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:38:18.665568 20435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:18.666796 20435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:18.667464 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:18.647527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.27.251.254:38551)
0808 18:38:18.647820 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.647825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.647841 (+ 16us) server_negotiation.cc:407] Connection header received
0808 18:38:18.647890 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.647896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.647954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.648046 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.649224 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.650032 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.657812 (+ 7780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:18.658199 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:18.658304 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:18.663678 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:18.663682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:18.663685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:18.663715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:18.665466 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.666020 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.666024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.666026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.666117 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:18.666473 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:18.666477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:18.666479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:18.666709 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:18.666832 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:18.667178 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 18:38:18.667295 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260808 18:38:18.679200 20267 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [term 1 FOLLOWER]: Refusing update from remote peer 52bc830f3c664488b6848d59bd145500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:38:18.679241 20348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [term 1 FOLLOWER]: Refusing update from remote peer 52bc830f3c664488b6848d59bd145500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:38:18.680047 20433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:38:18.682539 20433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:38:18.685046 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bc830f3c664488b6848d59bd145500. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.685194 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.687858 20414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bc830f3c664488b6848d59bd145500. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.687983 20414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.700412 20433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.700620 20433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:18.700737 20433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.700804 20433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:18.700927 20436 mvcc.cc:205] Tried to move back new op lower bound from 7316333767381168128 to 7316333767282704384. Current Snapshot: MvccSnapshot[applied={T|T < 7316333767381168128}]
I20260808 18:38:18.702683 20414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.702809 20414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.704100 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "52bc830f3c664488b6848d59bd145500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52bc830f3c664488b6848d59bd145500" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 38551 } } peers { permanent_uuid: "fd54916fdf834c7598d517beb55b9b8a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 43623 } } peers { permanent_uuid: "27786d89e5674e0d859aaeb90f4d71ec" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 38855 } } }
I20260808 18:38:18.704236 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:18.707702 20437 mvcc.cc:205] Tried to move back new op lower bound from 7316333767381168128 to 7316333767282704384. Current Snapshot: MvccSnapshot[applied={T|T < 7316333767381168128}]
I20260808 18:38:18.709873 20435 catalog_manager.cc:1347] Generated new cluster ID: ee87d7f429114c8c8302ea8057412213
I20260808 18:38:18.709942 20435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:18.746639 20435 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:18.748376 20435 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:18.762797 20435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52bc830f3c664488b6848d59bd145500: Generated new TSK 0
I20260808 18:38:18.763854 20435 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:38:18.798357 20378 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.795496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.27.251.252:38855)
0808 18:38:18.795657 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.795662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.797170 (+ 1508us) server_negotiation.cc:407] Connection header received
0808 18:38:18.797381 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.797385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.797436 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.797509 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.798213 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47012: BlockingRecv error: recv got EOF from 127.0.0.1:47012 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 18:38:18.800104 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.796343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.27.251.253:43623)
0808 18:38:18.796469 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.796474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.799170 (+ 2696us) server_negotiation.cc:407] Connection header received
0808 18:38:18.799348 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.799351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.799400 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.799451 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.799982 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45122: BlockingRecv error: recv got EOF from 127.0.0.1:45122 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 18:38:18.801942 20220 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.800731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.27.251.254:38551)
0808 18:38:18.800856 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.800860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.801150 (+ 290us) server_negotiation.cc:407] Connection header received
0808 18:38:18.801265 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.801270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.801324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.801442 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.801824 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52980: BlockingRecv error: recv got EOF from 127.0.0.1:52980 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 18:38:18.805960 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.804401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.27.251.253:43623)
0808 18:38:18.804596 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.804600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.805033 (+ 433us) server_negotiation.cc:407] Connection header received
0808 18:38:18.805245 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.805248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.805320 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.805370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.805866 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45126: BlockingRecv error: recv got EOF from 127.0.0.1:45126 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 18:38:18.810913 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.809549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.27.251.253:43623)
0808 18:38:18.809644 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.809647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.810105 (+ 458us) server_negotiation.cc:407] Connection header received
0808 18:38:18.810270 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.810273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.810317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.810364 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.810821 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45140: BlockingRecv error: recv got EOF from 127.0.0.1:45140 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 18:38:18.834055 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.824995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.27.251.253:43623)
0808 18:38:18.825122 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.825127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.832812 (+ 7685us) server_negotiation.cc:407] Connection header received
0808 18:38:18.833087 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.833092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.833156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.833217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.833923 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45146: BlockingRecv error: recv got EOF from 127.0.0.1:45146 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 18:38:18.842625 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.840748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.27.251.253:43623)
0808 18:38:18.840901 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.840907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.841556 (+ 649us) server_negotiation.cc:407] Connection header received
0808 18:38:18.841773 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.841777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.841832 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.841890 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.842498 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260808 18:38:18.855418 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.853422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.27.251.253:43623)
0808 18:38:18.853560 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.853573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.854279 (+ 706us) server_negotiation.cc:407] Connection header received
0808 18:38:18.854537 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.854541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.854601 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.854661 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.855295 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45164: BlockingRecv error: recv got EOF from 127.0.0.1:45164 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 18:38:18.876093 20299 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.874623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.27.251.253:43623)
0808 18:38:18.874756 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.874760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.875070 (+ 310us) server_negotiation.cc:407] Connection header received
0808 18:38:18.875236 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.875240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.875304 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.875411 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.875956 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45170: BlockingRecv error: recv got EOF from 127.0.0.1:45170 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260808 18:38:18.911726 20299 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.910468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.27.251.253:43623)
0808 18:38:18.910685 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.910689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.910707 (+ 18us) server_negotiation.cc:407] Connection header received
0808 18:38:18.910863 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.910865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.910918 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.910997 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.911569 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 18:38:18.979360 20299 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:18.978214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.27.251.253:43623)
0808 18:38:18.978359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:18.978363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:18.978469 (+ 106us) server_negotiation.cc:407] Connection header received
0808 18:38:18.978615 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:18.978617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:18.978676 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:18.978763 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:18.979261 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45178: BlockingRecv error: recv got EOF from 127.0.0.1:45178 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 18:38:19.111076 20299 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:19.109864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.27.251.253:43623)
0808 18:38:19.110004 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.110008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.110152 (+ 144us) server_negotiation.cc:407] Connection header received
0808 18:38:19.110255 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.110257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.110307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.110398 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:19.110932 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45192: BlockingRecv error: recv got EOF from 127.0.0.1:45192 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 18:38:19.370864 20299 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:19.369549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.27.251.253:43623)
0808 18:38:19.369817 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.369823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.369893 (+ 70us) server_negotiation.cc:407] Connection header received
0808 18:38:19.370082 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.370086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.370161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.370261 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:19.370757 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45196: BlockingRecv error: recv got EOF from 127.0.0.1:45196 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
I20260808 18:38:19.646068 20409 catalog_manager.cc:1259] Loaded cluster ID: ee87d7f429114c8c8302ea8057412213
I20260808 18:38:19.646157 20409 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: loading cluster ID for follower catalog manager: success
I20260808 18:38:19.647845 20409 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: acquiring CA information for follower catalog manager: success
I20260808 18:38:19.649049 20409 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd54916fdf834c7598d517beb55b9b8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:38:19.653934 20431 catalog_manager.cc:1259] Loaded cluster ID: ee87d7f429114c8c8302ea8057412213
I20260808 18:38:19.653992 20431 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: loading cluster ID for follower catalog manager: success
I20260808 18:38:19.655630 20431 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: acquiring CA information for follower catalog manager: success
I20260808 18:38:19.656862 20431 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27786d89e5674e0d859aaeb90f4d71ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:38:19.888973 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:19.885241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.27.251.253:43623)
0808 18:38:19.885503 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.885507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.885524 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:38:19.885637 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.885640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.885776 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.885880 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:19.886119 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.886681 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.887539 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.887758 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.887835 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:19.888120 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:19.888639 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:19.888729 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:19.888795 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 18:38:19.888843 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260808 18:38:19.895866 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:19.892267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.27.251.252:38855)
0808 18:38:19.892501 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.892547 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.892564 (+ 17us) server_negotiation.cc:407] Connection header received
0808 18:38:19.892654 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.892656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.892772 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.892874 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:19.893088 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.893597 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.894377 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.894689 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.894782 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:19.895034 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:19.895441 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:19.895531 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:19.895613 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 18:38:19.895660 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260808 18:38:19.903393 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:19.899280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.27.251.252:38855)
0808 18:38:19.899400 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.899405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.899728 (+ 323us) server_negotiation.cc:407] Connection header received
0808 18:38:19.900116 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.900120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.900169 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.900261 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:19.900637 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.901298 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902339 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.902563 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902679 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:19.902924 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:19.902995 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:19.903075 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:19.903202 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 18:38:19.903251 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 18:38:19.903801 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:19.899160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.27.251.253:43623)
0808 18:38:19.899320 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.899326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.899612 (+ 286us) server_negotiation.cc:407] Connection header received
0808 18:38:19.899844 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.899847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.899900 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.899983 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:19.900270 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.900913 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902354 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.902712 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902805 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:19.903292 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:19.903369 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:19.903414 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:19.903593 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 18:38:19.903649 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 18:38:19.904074 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:19.899099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.27.251.254:38551)
0808 18:38:19.899487 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:19.899492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:19.899587 (+ 95us) server_negotiation.cc:407] Connection header received
0808 18:38:19.899759 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:19.899762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:19.899915 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:19.900043 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:19.900295 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.901065 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902353 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:19.902738 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:19.902838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:19.903335 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:19.903673 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:19.903757 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:19.903849 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 18:38:19.903911 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":74,"threads_started":1}
I20260808 18:38:19.907589 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20145
I20260808 18:38:19.917542 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20224
I20260808 18:38:19.924968 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20302
2026-08-08T18:38:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3243 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 18:38:19 dist-test-slave-fd7l krb5kdc[20530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 18:38:19 dist-test-slave-fd7l krb5kdc[20530](info): set up 2 sockets
Aug 08 18:38:19 dist-test-slave-fd7l krb5kdc[20530](info): commencing operation
krb5kdc: starting...
W20260808 18:38:21.011997 28655 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/test-user.keytab.
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T18:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T18:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.251.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
Entry for principal kudu/127.27.251.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab.
I20260808 18:38:21.143424 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42647
--webserver_interface=127.27.251.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 18:38:21.257752 20546 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:21.258098 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:21.258179 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:21.258273 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:21.262658 20546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:21.262746 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:21.262794 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:21.262830 20546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:21.262862 20546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:21.268286 20546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.254.keytab
--principal=kudu/127.27.251.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.254:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.251.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:21.269388 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:21.270426 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:21.279019 20552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:21.279054 20554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:21.279014 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:21.279240 20546 server_base.cc:1034] running on GCE node
I20260808 18:38:21.279888 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:21.281055 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:21.282255 20546 hybrid_clock.cc:648] HybridClock initialized: now 1786214301282240 us; error 26 us; skew 500 ppm
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:21.285187 20546 init.cc:377] Logged in from keytab as kudu/127.27.251.254@KRBTEST.COM (short username kudu)
I20260808 18:38:21.286129 20546 webserver.cc:460] Webserver started at http://127.27.251.254:46063/ using document root <none> and password file <none>
I20260808 18:38:21.286710 20546 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:21.286806 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:21.287043 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:21.289211 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data/instance:
uuid: "42667c9428fd4338a562c2bb849ce594"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "0d461911f5056bc92c04ba0d7fb7166a"
server_key_iv: "5229d052ad9e2ca889ab24f029a8b7c6"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.289829 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance:
uuid: "42667c9428fd4338a562c2bb849ce594"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "0d461911f5056bc92c04ba0d7fb7166a"
server_key_iv: "5229d052ad9e2ca889ab24f029a8b7c6"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.293583 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:38:21.296006 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:21.297358 20546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:38:21.297519 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal
uuid: "42667c9428fd4338a562c2bb849ce594"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "0d461911f5056bc92c04ba0d7fb7166a"
server_key_iv: "5229d052ad9e2ca889ab24f029a8b7c6"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.297645 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:21.327775 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:21.331243 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:21.331471 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:21.339135 20546 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.254:42647
I20260808 18:38:21.339475 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.254:42647 every 8 connection(s)
I20260808 18:38:21.340185 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/data/info.pb
I20260808 18:38:21.340682 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20546
I20260808 18:38:21.340823 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-0/wal/instance
I20260808 18:38:21.341125 28655 external_mini_cluster.cc:1428] Setting key 276c333bdf2f41e3062e9027559d3c40
I20260808 18:38:21.343621 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:21.347703 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.251.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.253@KRBTEST.COM" created.
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:21.363332 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.350176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.27.251.254:42647)
0808 18:38:21.350646 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.350675 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.351036 (+ 361us) server_negotiation.cc:407] Connection header received
0808 18:38:21.352132 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.352136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.352323 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.352517 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.353446 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.354327 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.355308 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.355706 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.355877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.358245 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.358258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.358268 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.358290 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.360027 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.360565 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.360573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.360579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.360681 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.361019 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.361023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.361026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.361393 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.361542 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.361922 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.362200 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"spinlock_wait_cycles":1408,"thread_start_us":148,"threads_started":1}
I20260808 18:38:21.363332 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.350177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42647 (local address 127.0.0.1:60184)
0808 18:38:21.350873 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.350912 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.351815 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.352540 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.352545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.352989 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.353295 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.353311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.354455 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.354459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.355185 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.355194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.355865 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.355869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.355959 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.356605 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.356641 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.358102 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.360173 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.360182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.360191 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.360420 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.360787 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.360790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.360792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.360910 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.361531 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.361537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.361678 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.361847 (+ 169us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.362255 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":130,"spinlock_wait_cycles":34048,"thread_start_us":168,"threads_started":1}
I20260808 18:38:21.365945 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.366741 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.366424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43210)
0808 18:38:21.366581 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.366676 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:38:21.366997 20567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
I20260808 18:38:21.367894 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.367678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43222)
0808 18:38:21.367790 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.367842 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 18:38:21.368039 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:33819: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
Entry for principal kudu/127.27.251.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab.
I20260808 18:38:21.371750 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:33819
--webserver_interface=127.27.251.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:38:21.417347 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } attempt: 1
I20260808 18:38:21.418424 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.417995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43238)
0808 18:38:21.418206 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.418307 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 18:38:21.419373 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.419073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43252)
0808 18:38:21.419213 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.419287 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 18:38:21.419559 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:33819: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
I20260808 18:38:21.487773 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } attempt: 2
I20260808 18:38:21.488756 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.488336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43264)
0808 18:38:21.488564 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.488655 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 18:38:21.489562 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.489312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43266)
0808 18:38:21.489429 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.489484 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:38:21.489709 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.253:33819: Network error: Client connection negotiation failed: client connection to 127.27.251.253:33819: connect: Connection refused (error 111)
W20260808 18:38:21.493814 20627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:21.494073 20627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:21.494143 20627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 18:38:21.494212 20627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 18:38:21.498287 20627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:21.498356 20627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:21.498389 20627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:21.498426 20627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:21.498462 20627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:21.504060 20627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.253.keytab
--principal=kudu/127.27.251.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.253:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.251.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:21.505203 20627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:21.506228 20627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:21.514989 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:21.515045 20627 server_base.cc:1034] running on GCE node
W20260808 18:38:21.514981 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:21.514993 20637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:21.515887 20627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:21.517055 20627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:21.518226 20627 hybrid_clock.cc:648] HybridClock initialized: now 1786214301518205 us; error 31 us; skew 500 ppm
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:21.521404 20627 init.cc:377] Logged in from keytab as kudu/127.27.251.253@KRBTEST.COM (short username kudu)
I20260808 18:38:21.522368 20627 webserver.cc:460] Webserver started at http://127.27.251.253:37457/ using document root <none> and password file <none>
I20260808 18:38:21.523020 20627 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:21.523113 20627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:21.523351 20627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:21.525460 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data/instance:
uuid: "9192ae9b5b724f49bd0f4130fc75516a"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "2e8007afdfe49817ac667e4f3716b25b"
server_key_iv: "8317761585ccbb1fb8614dbcdb33ba0a"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.526046 20627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal/instance:
uuid: "9192ae9b5b724f49bd0f4130fc75516a"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "2e8007afdfe49817ac667e4f3716b25b"
server_key_iv: "8317761585ccbb1fb8614dbcdb33ba0a"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.529858 20627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 18:38:21.532315 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:21.533675 20627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260808 18:38:21.533829 20627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal
uuid: "9192ae9b5b724f49bd0f4130fc75516a"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "2e8007afdfe49817ac667e4f3716b25b"
server_key_iv: "8317761585ccbb1fb8614dbcdb33ba0a"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.533941 20627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:21.559904 20627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:21.563457 20627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:21.563686 20627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:21.571127 20627 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.253:33819
I20260808 18:38:21.571436 20697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.253:33819 every 8 connection(s)
I20260808 18:38:21.572139 20627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/data/info.pb
I20260808 18:38:21.575102 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:21.578907 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.579800 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20627
I20260808 18:38:21.579953 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-1/wal/instance
I20260808 18:38:21.580277 28655 external_mini_cluster.cc:1428] Setting key 04aa2d85f5ceb23d864c54651d3c9871
I20260808 18:38:21.580869 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } attempt: 3
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.251.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.251.252@KRBTEST.COM" created.
I20260808 18:38:21.594583 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.581294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43270)
0808 18:38:21.581489 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.581506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.581600 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.583341 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.583344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.583562 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.583685 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.583693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.585424 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.585428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.586226 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.586238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.587179 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.587184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.587297 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.587887 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.587901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.589664 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.592389 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.592392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.592394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.592663 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.593097 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.593100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.593102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.593198 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.593994 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.593997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.594114 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.594227 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.594379 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":3,"spinlock_wait_cycles":1408}
I20260808 18:38:21.594666 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.581034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.27.251.254:42647)
0808 18:38:21.581180 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.581187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.582089 (+ 902us) server_negotiation.cc:407] Connection header received
0808 18:38:21.583141 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.583147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.583210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.583306 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.584450 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.585033 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.586427 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.587163 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.587269 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.590399 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.590403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.590405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.590435 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.592067 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.592766 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.592771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.592773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.592876 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.593253 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.593259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.593262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.593508 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.593627 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.594161 (+ 534us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.594307 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":1280}
I20260808 18:38:21.596179 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.581419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42647 (local address 127.0.0.1:60188)
0808 18:38:21.581947 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.581998 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.582865 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.583500 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.583508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.583999 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.584310 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.584323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.585156 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.585159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.586294 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.586307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.587450 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.587456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.587563 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.588091 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.588115 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.590253 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.592213 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.592221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.592229 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.592628 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.592976 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.592978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.592980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.593143 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.593634 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.593644 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.593850 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.594423 (+ 573us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.595172 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":64,"spinlock_wait_cycles":3968,"thread_start_us":153,"threads_started":1}
I20260808 18:38:21.596186 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.581538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.27.251.253:33819)
0808 18:38:21.581997 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.582023 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.582064 (+ 41us) server_negotiation.cc:407] Connection header received
0808 18:38:21.582856 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.582909 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.583174 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.583354 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.583793 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.585284 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.586361 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.587032 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.587228 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.589883 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.589891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.589904 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.589932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.592281 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.592799 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.592806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.592810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.592993 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.593311 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.593314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.593316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.593850 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.594028 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.594327 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.595245 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":640,"server-negotiator.queue_time_us":376,"spinlock_wait_cycles":18688,"thread_start_us":173,"threads_started":1}
I20260808 18:38:21.597529 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.598505 20619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.599131 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.598900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48562)
0808 18:38:21.599019 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.599074 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 18:38:21.599769 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.599537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48566)
0808 18:38:21.599674 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.599725 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 18:38:21.599915 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:38:21.607113 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.598065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43286)
0808 18:38:21.598196 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.598211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.598278 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.598694 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.598700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.598907 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.599043 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.599056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.600366 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.600369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.601074 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.601084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.601725 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.601728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.601794 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.602382 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.602394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.603928 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.605665 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.605667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.605669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.605861 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.606140 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.606142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.606143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.606217 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.606644 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.606648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.606737 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.606832 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.606932 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 18:38:21.607141 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.598142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.27.251.253:33819)
0808 18:38:21.598239 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.598243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.598263 (+ 20us) server_negotiation.cc:407] Connection header received
0808 18:38:21.598376 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.598382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.598435 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.598515 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.599270 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.600217 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.601199 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.601601 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.601732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.604057 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.604060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.604062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.604084 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.605523 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.605975 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.605978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.605979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.606053 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.606297 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.606299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.606300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.606524 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.606602 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.606890 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.606985 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260808 18:38:21.607985 20698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.608711 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.608402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48580)
0808 18:38:21.608561 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.608660 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 18:38:21.609269 20646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
I20260808 18:38:21.610203 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.609997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48588)
0808 18:38:21.610108 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.610156 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 18:38:21.610379 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
Entry for principal kudu/127.27.251.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab.
I20260808 18:38:21.612433 28655 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5xHY3t/build/debug/bin/kudu
/tmp/dist-test-task5xHY3t/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:36253
--webserver_interface=127.27.251.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.251.212:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 18:38:21.631114 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 1
I20260808 18:38:21.632025 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.631682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48604)
0808 18:38:21.631856 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.631930 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 18:38:21.632855 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.632608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48610)
0808 18:38:21.632721 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.632775 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 18:38:21.633019 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
I20260808 18:38:21.659698 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 1
I20260808 18:38:21.660650 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.660279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48616)
0808 18:38:21.660437 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.660557 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 18:38:21.661446 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.661233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48626)
0808 18:38:21.661343 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.661389 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:38:21.661599 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
I20260808 18:38:21.708222 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 2
I20260808 18:38:21.709133 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.708779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48630)
0808 18:38:21.708959 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.709036 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260808 18:38:21.709971 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.709716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48636)
0808 18:38:21.709837 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.709898 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 18:38:21.710135 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
W20260808 18:38:21.729460 20706 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 18:38:21.729775 20706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 18:38:21.729825 20706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260808 18:38:21.729775 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 2
W20260808 18:38:21.729882 20706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 18:38:21.730589 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.730289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48644)
0808 18:38:21.730440 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.730510 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 18:38:21.731334 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.731103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48650)
0808 18:38:21.731233 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.731284 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 18:38:21.731482 20698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
W20260808 18:38:21.734203 20706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 18:38:21.734287 20706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 18:38:21.734326 20706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 18:38:21.734385 20706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 18:38:21.734426 20706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 18:38:21.740144 20706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.251.212:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.251.254:42647,127.27.251.253:33819,127.27.251.252:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/krb5kdc/kudu_127.27.251.252.keytab
--principal=kudu/127.27.251.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.251.252:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.251.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 17:55:42 UTC on ccycloud.quasar-domehl.root.comops.site
build id 4575373
I20260808 18:38:21.741355 20706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 18:38:21.742450 20706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 18:38:21.751230 20712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:21.751266 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 18:38:21.751341 20711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 18:38:21.751782 20706 server_base.cc:1034] running on GCE node
I20260808 18:38:21.752305 20706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 18:38:21.753455 20706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 18:38:21.754618 20706 hybrid_clock.cc:648] HybridClock initialized: now 1786214301754587 us; error 44 us; skew 500 ppm
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 18:38:21.757534 20706 init.cc:377] Logged in from keytab as kudu/127.27.251.252@KRBTEST.COM (short username kudu)
I20260808 18:38:21.758381 20706 webserver.cc:460] Webserver started at http://127.27.251.252:46647/ using document root <none> and password file <none>
I20260808 18:38:21.758973 20706 fs_manager.cc:359] Metadata directory not provided
I20260808 18:38:21.759037 20706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 18:38:21.759209 20706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 18:38:21.761484 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data/instance:
uuid: "ff4ebc6ab7b1497ba89cea4f9117a960"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "fd17a9779cea25f613406a74a15a3bec"
server_key_iv: "080ca07fc5b617df39affc51faa10a7c"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.762004 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal/instance:
uuid: "ff4ebc6ab7b1497ba89cea4f9117a960"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "fd17a9779cea25f613406a74a15a3bec"
server_key_iv: "080ca07fc5b617df39affc51faa10a7c"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.765759 20706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 18:38:21.768083 20723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 18:38:21.769411 20706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260808 18:38:21.769556 20706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data,/tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal
uuid: "ff4ebc6ab7b1497ba89cea4f9117a960"
format_stamp: "Formatted at 2026-08-08 18:38:21 on dist-test-slave-fd7l"
server_key: "fd17a9779cea25f613406a74a15a3bec"
server_key_iv: "080ca07fc5b617df39affc51faa10a7c"
server_key_version: "encryptionkey@0"
I20260808 18:38:21.769677 20706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 18:38:21.800050 20706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 18:38:21.803526 20706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 18:38:21.803743 20706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 18:38:21.809324 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 3
I20260808 18:38:21.810163 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.809868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48656)
0808 18:38:21.809988 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.810057 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 18:38:21.810974 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.810719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48662)
0808 18:38:21.810832 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.810879 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 18:38:21.811184 20619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.251.252:36253: Network error: Client connection negotiation failed: client connection to 127.27.251.252:36253: connect: Connection refused (error 111)
I20260808 18:38:21.811393 20706 rpc_server.cc:307] RPC server started. Bound to: 127.27.251.252:36253
I20260808 18:38:21.811698 20778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.251.252:36253 every 8 connection(s)
I20260808 18:38:21.812398 20706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/data/info.pb
I20260808 18:38:21.815156 20779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 18:38:21.818955 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.819875 28655 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5xHY3t/build/debug/bin/kudu as pid 20706
I20260808 18:38:21.820004 28655 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5xHY3t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786214007390461-28655-0/minicluster-data/master-2/wal/instance
I20260808 18:38:21.820308 28655 external_mini_cluster.cc:1428] Setting key d73d835db6c00fdc396a405e8b7011c6
I20260808 18:38:21.822665 20698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 3
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.253@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:38:21.834447 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.823141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48686)
0808 18:38:21.823328 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.823342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.823420 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.824146 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.824149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.824348 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.824437 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.824443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.825552 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.825556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.826374 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.826385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.827276 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.827283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.827390 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.827822 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.827839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.829483 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.832072 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.832075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.832078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.832295 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.832755 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.832760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.832763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.832859 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.833877 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.833880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.833993 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.834110 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.834272 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260808 18:38:21.835847 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.820910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.27.251.254:42647)
0808 18:38:21.821023 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.821027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.822106 (+ 1079us) server_negotiation.cc:407] Connection header received
0808 18:38:21.822901 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.822907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.822971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.823016 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.824294 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.825098 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.826233 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.826514 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.826619 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.830936 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.830940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.830943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.830979 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.832836 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.834252 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.834256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.834258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.834348 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.834828 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.834832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.834833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.835054 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.835148 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.835586 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.835712 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:38:21.836200 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.821402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.254:42647 (local address 127.0.0.1:60196)
0808 18:38:21.821937 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.822012 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.822649 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.823269 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.823274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.823741 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.824143 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.824159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.825237 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.825241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.826090 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.826099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.826658 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.826661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.827034 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.827666 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.827681 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.830736 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.832957 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.832969 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.832976 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.834152 (+ 1176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.834454 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.834460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.834465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.834713 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.835231 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.835237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.835352 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.835496 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.835671 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":98,"thread_start_us":194,"threads_started":1}
I20260808 18:38:21.836203 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.823631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.27.251.252:36253)
0808 18:38:21.823896 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.823906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.823916 (+ 10us) server_negotiation.cc:407] Connection header received
0808 18:38:21.823943 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.823945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.824000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.824098 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.824572 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.825453 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.826499 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.827141 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.827271 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.829627 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.829634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.829647 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.829678 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.831936 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.832452 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.832459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.832467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.832649 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.833162 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.833167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.833169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.833757 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.833914 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.834793 (+ 879us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.835117 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":879,"server-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
I20260808 18:38:21.836206 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.821401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.27.251.252:36253)
0808 18:38:21.821820 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.821843 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.821888 (+ 45us) server_negotiation.cc:407] Connection header received
0808 18:38:21.822638 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.822655 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.822935 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.823168 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.824046 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.825371 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.826532 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.827132 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.827252 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.830850 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.830853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.830855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.830878 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.833286 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.833748 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.833751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.833752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.833824 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.834120 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.834124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.834126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.834357 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.834531 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.835230 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.835335 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":258,"thread_start_us":125,"threads_started":1}
I20260808 18:38:21.837528 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.840884 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: Bootstrap starting.
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
I20260808 18:38:21.843891 20698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: Neither blocks nor log segments found. Creating new log.
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.254@KRBTEST.COM
I20260808 18:38:21.844954 20698 log.cc:824] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:21.847208 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.838078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.27.251.253:33819)
0808 18:38:21.838223 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.838228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.838279 (+ 51us) server_negotiation.cc:407] Connection header received
0808 18:38:21.838466 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.838470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.838524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.838600 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.839077 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.839772 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.840876 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.841180 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.841292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.843361 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.843365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.843367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.843392 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.845196 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.845741 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.845745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.845746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.845839 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.846212 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.846215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.846218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.846455 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.846526 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.846915 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.847031 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 18:38:21.847436 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: No bootstrap required, opened a new log
I20260808 18:38:21.847682 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.838020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.253:33819 (local address 127.0.0.1:43296)
0808 18:38:21.838226 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.838240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.838319 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.838616 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.838619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.838819 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.838951 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.838958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.839899 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.839905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.840756 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.840767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.841305 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.841310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.841407 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.841853 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.841867 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.843232 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.845332 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.845338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.845343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.845607 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.845954 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.845958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.845959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.846066 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.846601 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.846604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.846709 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.847394 (+ 685us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.847542 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 18:38:21.848426 20779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } has no permanent_uuid. Determining permanent_uuid...
I20260808 18:38:21.849023 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.839226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.27.251.254:42647)
0808 18:38:21.839333 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.839337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.839349 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:38:21.839429 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.839432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.839470 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.839561 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.840254 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.840978 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.841997 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.842379 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.842490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.845233 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.845236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.845239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.845271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.847023 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.847611 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.847614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.847615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.847693 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.848055 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.848058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.848059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.848293 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.848559 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.848744 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.848875 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 18:38:21.851661 20698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:21.851886 20698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9192ae9b5b724f49bd0f4130fc75516a, State: Initialized, Role: FOLLOWER
I20260808 18:38:21.852246 20698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:21.852803 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:21.853034 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:21.853706 20698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.251.253@KRBTEST.COM
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.252@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
W20260808 18:38:21.859746 20798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:21.859879 20798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:21.860174 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.850311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.27.251.253:33819)
0808 18:38:21.850417 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.850421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.850434 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:38:21.850466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.850468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.850508 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.850591 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.851157 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.851831 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.852894 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.853197 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.853298 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.856278 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.856281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.856284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.856307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.857934 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.858447 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.858450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.858452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.858529 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.858847 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.858852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.858854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.859047 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.859731 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.859813 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.860025 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 18:38:21.858373 20627 master_runner.cc:186] Error getting master registration for 127.27.251.254:42647: OK, instance_id { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" instance_seqno: 1786214301337048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:38:21.862068 20627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:38:21.864254 28655 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 18:38:21.865022 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.849159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.27.251.252:36253)
0808 18:38:21.852072 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.852076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.852087 (+ 11us) server_negotiation.cc:407] Connection header received
0808 18:38:21.852113 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.852115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.852155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.852246 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.852725 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.853386 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.854300 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.854519 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.854639 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.861027 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.861031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.861034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.861059 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.863112 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.863653 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.863657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.863659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.863753 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.864053 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.864055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.864057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.864336 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.864429 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.864724 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.864859 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2838}
I20260808 18:38:21.865453 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.848952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48696)
0808 18:38:21.849364 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.849378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.849454 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.852256 (+ 2802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.852259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.852455 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.852568 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.852593 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.853493 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.853499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.854170 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.854180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.854779 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.854784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.854869 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.855337 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.855351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.860866 (+ 5515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.863248 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.863252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.863254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.863518 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.863862 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.863866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.863868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.863955 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.864460 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.864462 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.864573 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.865209 (+ 636us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.865330 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339}
I20260808 18:38:21.865767 20591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "42667c9428fd4338a562c2bb849ce594"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60198
I20260808 18:38:21.868242 20779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: Bootstrap starting.
I20260808 18:38:21.871058 20779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:21.871968 20779 log.cc:824] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:21.874320 20779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: No bootstrap required, opened a new log
I20260808 18:38:21.877817 20779 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:21.878047 20779 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4ebc6ab7b1497ba89cea4f9117a960, State: Initialized, Role: FOLLOWER
I20260808 18:38:21.878381 20779 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:21.878873 20801 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:21.879004 20801 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:21.879650 20779 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 18:38:21.885896 20813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 18:38:21.885982 20813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:21.883935 20706 master_runner.cc:186] Error getting master registration for 127.27.251.254:42647: OK, instance_id { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" instance_seqno: 1786214301337048 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 18:38:21.887651 20706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:38:21.975373 20619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } attempt: 4
Aug 08 18:38:21 dist-test-slave-fd7l krb5kdc[20530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786214301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.251.254@KRBTEST.COM for kudu/127.27.251.252@KRBTEST.COM
I20260808 18:38:21.986369 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.976033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.251.252:36253 (local address 127.0.0.1:48708)
0808 18:38:21.976381 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0808 18:38:21.976397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 18:38:21.976503 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 18:38:21.976867 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 18:38:21.976870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 18:38:21.977095 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 18:38:21.977220 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.977227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.978333 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.978337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.979199 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 18:38:21.979210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.980013 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.980018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 18:38:21.980115 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.980780 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 18:38:21.980799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 18:38:21.982476 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 18:38:21.984662 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.984666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.984668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.984925 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.985250 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 18:38:21.985253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 18:38:21.985255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 18:38:21.985349 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 18:38:21.985812 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 18:38:21.985818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 18:38:21.985931 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 18:38:21.986044 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 18:38:21.986189 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260808 18:38:21.986730 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:21.976416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.27.251.252:36253)
0808 18:38:21.976585 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:21.976590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:21.976611 (+ 21us) server_negotiation.cc:407] Connection header received
0808 18:38:21.976671 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:21.976676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:21.976735 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:21.976856 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:21.977370 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.978215 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.979341 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:21.979715 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:21.979833 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:21.982627 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:21.982632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:21.982635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:21.982665 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:21.984458 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.985041 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.985045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.985047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.985152 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:21.985457 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:21.985461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:21.985462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:21.985711 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:21.985825 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:21.986133 (+ 308us) server_negotiation.cc:299] Negotiation successful
0808 18:38:21.986272 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 18:38:21.990464 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: Bootstrap starting.
I20260808 18:38:21.993537 20619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: Neither blocks nor log segments found. Creating new log.
I20260808 18:38:21.994478 20619 log.cc:824] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: Log is configured to *not* fsync() on all Append() calls
I20260808 18:38:21.997247 20619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: No bootstrap required, opened a new log
I20260808 18:38:22.000537 20619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:22.000816 20619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42667c9428fd4338a562c2bb849ce594, State: Initialized, Role: FOLLOWER
I20260808 18:38:22.001209 20619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:22.001926 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.002272 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:22.003057 20619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 18:38:22.003142 20591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 18:38:22.003283 20591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:22.004748 20591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:22.006635 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42667c9428fd4338a562c2bb849ce594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960"
I20260808 18:38:22.006933 20752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:22.008890 20672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42667c9428fd4338a562c2bb849ce594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9192ae9b5b724f49bd0f4130fc75516a"
I20260808 18:38:22.009171 20672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [term 0 FOLLOWER]: Advancing to term 1
I20260808 18:38:22.010641 20672 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42667c9428fd4338a562c2bb849ce594 in term 1.
W20260808 18:38:22.011044 20828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 18:38:22.013945 20752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42667c9428fd4338a562c2bb849ce594 in term 1.
I20260808 18:38:22.012708 20567 leader_election.cc:302] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42667c9428fd4338a562c2bb849ce594, 9192ae9b5b724f49bd0f4130fc75516a; no voters:
I20260808 18:38:22.005780 20591 leader_election.cc:288] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [CANDIDATE]: Term 1 election: Requested vote from peers 9192ae9b5b724f49bd0f4130fc75516a (127.27.251.253:33819), ff4ebc6ab7b1497ba89cea4f9117a960 (127.27.251.252:36253)
I20260808 18:38:22.016844 20816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 18:38:22.017120 20816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [term 1 LEADER]: Becoming Leader. State: Replica: 42667c9428fd4338a562c2bb849ce594, State: Running, Role: LEADER
W20260808 18:38:22.017352 20828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 18:38:22.017735 20816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } }
I20260808 18:38:22.019531 20546 master_runner.cc:263] No leader master found from master 42667c9428fd4338a562c2bb849ce594
I20260808 18:38:22.019614 20546 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 18:38:22.019726 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42667c9428fd4338a562c2bb849ce594. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.019855 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:22.024581 20836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 18:38:22.025692 20836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 18:38:22.049072 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:22.021337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.27.251.254:42647)
0808 18:38:22.021450 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.021455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.034086 (+ 12631us) server_negotiation.cc:407] Connection header received
0808 18:38:22.034303 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.034307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.034364 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.034428 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.035526 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.036268 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.037558 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.037910 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.038006 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:22.044948 (+ 6942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:22.044953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:22.044956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:22.044988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:22.046924 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.047472 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.047478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.047481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.047573 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.047889 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.047893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.047909 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.048167 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:22.048292 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:22.048721 (+ 429us) server_negotiation.cc:299] Negotiation successful
0808 18:38:22.048865 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 18:38:22.055212 20730 tablet.cc:2369] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:38:22.056631 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:22.020451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.27.251.252:36253)
0808 18:38:22.020606 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.020613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.028726 (+ 8113us) server_negotiation.cc:407] Connection header received
0808 18:38:22.028984 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.028988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.029048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.029114 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.030318 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.031067 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.032604 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.032966 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.033062 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:22.052240 (+ 19178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:22.052245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:22.052248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:22.052282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:22.054257 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.054960 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.054964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.054966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.055065 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.055454 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.055458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.055460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.055723 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:22.055850 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:22.056268 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 18:38:22.056401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 18:38:22.064874 20752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [term 1 FOLLOWER]: Refusing update from remote peer 42667c9428fd4338a562c2bb849ce594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 18:38:22.065389 20650 tablet.cc:2369] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 18:38:22.065977 20834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:38:22.070641 20801 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42667c9428fd4338a562c2bb849ce594. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.070822 20801 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:22.074055 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:22.022048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.27.251.253:33819)
0808 18:38:22.022199 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.022205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.039005 (+ 16800us) server_negotiation.cc:407] Connection header received
0808 18:38:22.039205 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.039209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.039266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.039323 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.040351 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.041111 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.042275 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:22.042611 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:22.042686 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:22.058221 (+ 15535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 18:38:22.058225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 18:38:22.058229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 18:38:22.058264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 18:38:22.060252 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.065004 (+ 4752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.065008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.065010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.065114 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 18:38:22.072791 (+ 7677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 18:38:22.072795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 18:38:22.072797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 18:38:22.073066 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 18:38:22.073195 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:22.073717 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 18:38:22.073879 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 18:38:22.076869 20672 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [term 1 FOLLOWER]: Refusing update from remote peer 42667c9428fd4338a562c2bb849ce594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 18:38:22.077529 20834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 18:38:22.079027 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.079160 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:22.079291 20840 mvcc.cc:205] Tried to move back new op lower bound from 7316333781247569920 to 7316333781068046336. Current Snapshot: MvccSnapshot[applied={T|T < 7316333781247569920}]
I20260808 18:38:22.079501 20801 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.079634 20801 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:22.079872 20841 mvcc.cc:205] Tried to move back new op lower bound from 7316333781247569920 to 7316333781068046336. Current Snapshot: MvccSnapshot[applied={T|T < 7316333781247569920}]
I20260808 18:38:22.082067 20836 catalog_manager.cc:1347] Generated new cluster ID: 59b5d74d40f34f3babc8c2de4fd483db
I20260808 18:38:22.082140 20836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 18:38:22.083428 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42667c9428fd4338a562c2bb849ce594. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.083570 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:22.086170 20834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.086299 20834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594 [sys.catalog]: This master's current role is: LEADER
I20260808 18:38:22.087481 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42667c9428fd4338a562c2bb849ce594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42667c9428fd4338a562c2bb849ce594" member_type: VOTER last_known_addr { host: "127.27.251.254" port: 42647 } } peers { permanent_uuid: "9192ae9b5b724f49bd0f4130fc75516a" member_type: VOTER last_known_addr { host: "127.27.251.253" port: 33819 } } peers { permanent_uuid: "ff4ebc6ab7b1497ba89cea4f9117a960" member_type: VOTER last_known_addr { host: "127.27.251.252" port: 36253 } } }
I20260808 18:38:22.087605 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 18:38:22.096943 20836 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 18:38:22.098263 20836 catalog_manager.cc:1504] Loading token signing keys...
I20260808 18:38:22.110670 20836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42667c9428fd4338a562c2bb849ce594: Generated new TSK 0
I20260808 18:38:22.111572 20836 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 18:38:22.137180 20784 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.135593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.27.251.252:36253)
0808 18:38:22.135736 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.135741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.135936 (+ 195us) server_negotiation.cc:407] Connection header received
0808 18:38:22.136135 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.136140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.136204 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.136261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.136987 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48726: BlockingRecv error: recv got EOF from 127.0.0.1:48726 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 18:38:22.137311 20622 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.135604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.27.251.254:42647)
0808 18:38:22.135734 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.135739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.136212 (+ 473us) server_negotiation.cc:407] Connection header received
0808 18:38:22.136412 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.136416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.136483 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.136578 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.137187 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60222: BlockingRecv error: recv got EOF from 127.0.0.1:60222 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260808 18:38:22.138053 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.136229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.27.251.253:33819)
0808 18:38:22.136361 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.136367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.136888 (+ 521us) server_negotiation.cc:407] Connection header received
0808 18:38:22.137073 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.137076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.137125 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.137180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.137958 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43320: BlockingRecv error: recv got EOF from 127.0.0.1:43320 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 18:38:22.141517 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.140301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.27.251.253:33819)
0808 18:38:22.140419 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.140424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.140600 (+ 176us) server_negotiation.cc:407] Connection header received
0808 18:38:22.140810 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.140813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.140889 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.141001 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.141416 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43324: BlockingRecv error: recv got EOF from 127.0.0.1:43324 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 18:38:22.145988 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.145046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.27.251.253:33819)
0808 18:38:22.145185 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.145188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.145201 (+ 13us) server_negotiation.cc:407] Connection header received
0808 18:38:22.145242 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.145244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.145283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.145404 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.145872 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43328: BlockingRecv error: recv got EOF from 127.0.0.1:43328 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 18:38:22.150913 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.149771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.27.251.253:33819)
0808 18:38:22.149879 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.149884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.150073 (+ 189us) server_negotiation.cc:407] Connection header received
0808 18:38:22.150208 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.150210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.150249 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.150314 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.150809 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43332: BlockingRecv error: recv got EOF from 127.0.0.1:43332 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260808 18:38:22.157977 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.156758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.27.251.253:33819)
0808 18:38:22.156916 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.156919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.157073 (+ 154us) server_negotiation.cc:407] Connection header received
0808 18:38:22.157181 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.157183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.157222 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.157287 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.157874 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43334: BlockingRecv error: recv got EOF from 127.0.0.1:43334 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 18:38:22.169749 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.168229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.27.251.253:33819)
0808 18:38:22.168379 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.168386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.168592 (+ 206us) server_negotiation.cc:407] Connection header received
0808 18:38:22.168821 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.168825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.168917 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.169037 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.169609 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260808 18:38:22.189715 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.188371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.27.251.253:33819)
0808 18:38:22.188577 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.188583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.188738 (+ 155us) server_negotiation.cc:407] Connection header received
0808 18:38:22.188850 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.188854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.188921 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.189042 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.189576 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43346: BlockingRecv error: recv got EOF from 127.0.0.1:43346 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 18:38:22.225309 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.223979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.27.251.253:33819)
0808 18:38:22.224161 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.224166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.224298 (+ 132us) server_negotiation.cc:407] Connection header received
0808 18:38:22.224486 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.224490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.224585 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.224686 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.225190 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43352: BlockingRecv error: recv got EOF from 127.0.0.1:43352 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260808 18:38:22.293057 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.291862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.27.251.253:33819)
0808 18:38:22.292010 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.292014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.292174 (+ 160us) server_negotiation.cc:407] Connection header received
0808 18:38:22.292285 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.292287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.292334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.292418 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.292913 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43358: BlockingRecv error: recv got EOF from 127.0.0.1:43358 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 18:38:22.424852 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.423573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.27.251.253:33819)
0808 18:38:22.423743 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.423747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.424020 (+ 273us) server_negotiation.cc:407] Connection header received
0808 18:38:22.424112 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.424114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.424166 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.424239 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.424704 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43368: BlockingRecv error: recv got EOF from 127.0.0.1:43368 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 18:38:22.684799 20701 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 18:38:22.683456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.27.251.253:33819)
0808 18:38:22.683621 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:22.683626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:22.683753 (+ 127us) server_negotiation.cc:407] Connection header received
0808 18:38:22.683953 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:22.683956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:22.684027 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:22.684132 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 18:38:22.684667 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43382: BlockingRecv error: recv got EOF from 127.0.0.1:43382 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 18:38:22.861346 20798 catalog_manager.cc:1259] Loaded cluster ID: 59b5d74d40f34f3babc8c2de4fd483db
I20260808 18:38:22.861478 20798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: loading cluster ID for follower catalog manager: success
I20260808 18:38:22.863188 20798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: acquiring CA information for follower catalog manager: success
I20260808 18:38:22.864378 20798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9192ae9b5b724f49bd0f4130fc75516a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:38:22.887451 20813 catalog_manager.cc:1259] Loaded cluster ID: 59b5d74d40f34f3babc8c2de4fd483db
I20260808 18:38:22.887534 20813 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: loading cluster ID for follower catalog manager: success
I20260808 18:38:22.889376 20813 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: acquiring CA information for follower catalog manager: success
I20260808 18:38:22.890563 20813 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff4ebc6ab7b1497ba89cea4f9117a960: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 18:38:23.203717 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:23.199188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.27.251.253:33819)
0808 18:38:23.199558 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:23.199564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:23.199690 (+ 126us) server_negotiation.cc:407] Connection header received
0808 18:38:23.199885 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:23.199891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:23.200054 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:23.200191 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:23.200478 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.201058 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.202069 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.202313 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.202431 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:23.202773 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:23.203259 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:23.203401 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:23.203487 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 18:38:23.203547 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":79,"threads_started":1}
I20260808 18:38:23.211010 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:23.207218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.27.251.252:36253)
0808 18:38:23.207465 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:23.207469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:23.207483 (+ 14us) server_negotiation.cc:407] Connection header received
0808 18:38:23.207648 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:23.207650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:23.207775 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:23.207874 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:23.208117 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.208730 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.209654 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.209871 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.209946 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:23.210238 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:23.210654 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:23.210746 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:23.210816 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 18:38:23.210888 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260808 18:38:23.218317 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:23.214715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.27.251.253:33819)
0808 18:38:23.214845 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:23.214855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:23.214867 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:38:23.214910 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:23.214912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:23.214950 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:23.215010 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:23.215274 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.216012 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.217196 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.217454 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.217567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:23.217849 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:23.217930 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:23.218011 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:23.218089 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 18:38:23.218165 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 18:38:23.218901 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:23.214294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.27.251.252:36253)
0808 18:38:23.214408 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:23.214411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:23.214648 (+ 237us) server_negotiation.cc:407] Connection header received
0808 18:38:23.215517 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:23.215520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:23.215570 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:23.215645 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:23.215903 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.216559 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.217683 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.217999 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.218074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:23.218441 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:23.218507 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:23.218583 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:23.218735 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 18:38:23.218788 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 18:38:23.219456 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 18:38:23.214182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.27.251.254:42647)
0808 18:38:23.214822 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0808 18:38:23.214832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 18:38:23.214844 (+ 12us) server_negotiation.cc:407] Connection header received
0808 18:38:23.214923 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 18:38:23.214925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 18:38:23.215090 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 18:38:23.215234 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 18:38:23.215465 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.216250 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.217247 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 18:38:23.217593 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 18:38:23.217694 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 18:38:23.218022 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 18:38:23.218515 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 18:38:23.218595 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 18:38:23.219297 (+ 702us) server_negotiation.cc:299] Negotiation successful
0808 18:38:23.219349 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":82,"threads_started":1}
I20260808 18:38:23.227074 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20546
I20260808 18:38:23.236407 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20627
I20260808 18:38:23.244326 28655 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5xHY3t/build/debug/bin/kudu with pid 20706
2026-08-08T18:38:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3319 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6563 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (295849 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST